Shell Scripting (bash, zsh, POSIX sh)

Writing, reviewing, and debugging shell scripts that run unattended: strict mode, quoting, arrays, traps, portability, shellcheck, and bats testing.

Choosing the Shell

| Target | Interpreter | Why | |---|---|---| | Interactive use, complex automation | bash 5.x | Arrays, [[ ]], mapfile, set -o pipefail | | macOS login shell / scripts by users | zsh 5.x | Default on macOS; bash there is 3.2 (2007) | | Must run everywhere (Alpine, BusyBox, dash, CI) | POSIX sh | No arrays, no [[ ]], no pipefail in strict POSIX | | Ownership of behavior matters | Pin #!/usr/bin/env bash | Avoid /bin/sh ambiguity; document the floor | | Logic is growing past ~200 lines | Python or Go | Shell lacks types, tests, and data structures |

• #!/usr/bin/env bash picks up the user's PATH; #!/bin/bash is deterministic but wrong on systems where bash lives elsewhere (NixOS, FreeBSD).
• zsh is not bash: arrays are 1-indexed, $array expands all elements, no word splitting by default, setopt differs. Run emulate -L sh or simply write bash when portability across the two is required.
• checkbashisms (Debian devscripts) flags bash-only constructs in sh scripts.

Strict Mode

#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'

| Option | Effect | Caveat | |---|---|---| | -e / errexit | Exit on unchecked failure | Ignored in if, while, &&, \|\| contexts | | -u / nounset | Error on unset variable | "$@" with no args is fine; $1 is not | | -o pipefail | Pipeline fails on any stage | Not POSIX; bash/zsh/ksh only | | -E / errtrace | ERR trap inherited by functions/subshells | Required for useful ERR traps | | IFS=$'\n\t' | Safer word splitting | Omit tab only if you need multi-line reads |

set -e caveats worth memorizing:

• Failure inside command substitution aborts only with set -e in effect for the subshell; local x=$(false) returns 0 because local masks the status. Split it: local x; x=$(false).
• Arithmetic ((i++)) returns 1 when the result is 0 and can abort the script. Use i=$((i + 1)) or ((++i)).
• grep pattern file returns 1 when there are no matches; under -e that is fatal. Write if grep -q ...; then or grep ... || true deliberately.
• read returning 1 at EOF aborts loops if unguarded: while read -r line; do ...; done < file fails on the loop condition — use while IFS= read -r line || [[ -n $line ]].
• A function that is meant to return non-zero will kill the caller under -e unless the call site handles it (if ! f; then).

Quoting and Expansion

printf '%s\n' "$name" "$@"              # always quote
[[ $path == "$prefix"/* ]] || exit 1    # [[ ]] suppresses splitting, still quote for clarity
out=$(cmd --flag)                       # capture stdout, don't quote $( ) inner args
files=( "$dir"/*.txt )                  # glob into array without word-splitting surprises
cp -- "$src" "$dst"                     # -- ends option parsing for hostile paths

| Form | Meaning | Never do | |---|---|---| | "$var" | Expand, no splitting/globbing | $var unquoted | | "${var:-default}" | Default if unset/empty | $var with set -u | | "${var:?message}" | Abort with message if unset | Silent empty expansion | | "${#var}" | Length | expr length | | "${var##*/}" / "${var%%/*}" | Basename / dirname without forks | basename/dirname in hot loops | | "${var//pat/repl}" | Replace all | sed for simple substitution | | "$@" | All positional params, one word each | $* or "$*" unless deliberate | | ${!prefix@} | Variable indirection | eval |

• Bash expansion order: brace → tilde → parameter/variable → command substitution → arithmetic → word splitting → pathname expansion. Quote to suppress the last two.
• printf over echo: echo -e is not portable, echo mangles backslashes, and printf '%s\n' is explicit. Use printf -v var '%s' ... to assign.
• IFS= read -r line preserves backslashes and leading whitespace; without -r, backslash escapes are processed.

Arrays (bash)

declare -a files=("a b.txt" "c.txt")     # indexed
declare -A seen=([a]=1 [b]=2)            # associative
files+=("d.txt")
for f in "${files[@]}"; do printf '%s\n' "$f"; done
mapfile -t lines < <(journalctl -n 50 -o cat)   # safe line reader
keys=("${!seen[@]}")                     # assoc keys
[[ -v seen[a] ]] && printf 'present\n'   # membership

• ${arr[@]} unquoted splits and globs; ${arr[*]} joins with IFS[0].
• Sparse arrays iterate indices; ${#arr[@]} is the count, ${#arr[0]} is a length.
• Bash 4.3+ supports namerefs: declare -n ref=$1, but prefer explicit variables in scripts meant to be read by others.
• zsh differs: "${(@)arr}" semantics, $arr[1] indexing, no associative declare -A (use typeset -A). Prefer bash for array-heavy scripts.

Functions, Exit Codes, Traps

die() { printf 'error: %s\n' "$*" >&2; exit 1; }

cleanup() {
  local rc=$?
  [[ -n ${tmpdir:-} ]] && rm -rf -- "$tmpdir"
  exit "$rc"
}
on_err() { printf 'FAIL line %d: %s\n' "$1" "$2" >&2; }
trap cleanup EXIT
trap 'on_err "$LINENO" "$BASH_COMMAND"' ERR
trap 'echo interrupted; exit 130' INT TERM HUP

tmpdir=$(mktemp -d) || die "mktemp failed"

• EXIT traps run on every exit path, including exit in nested functions; keep cleanup idempotent and never set -e inside it.
• Signal traps should exit with 128+N (130 for SIGINT, 143 for SIGTERM) so supervisors see the right status. In a container, PID 1 must forward signals or docker stop will SIGKILL after the grace period.
• trap - EXIT removes a handler; re-entry from cleanup is a classic source of double deletes.
• Function-local variables require local (declare inside a function is local to it, but declare in bash without a function is global). Arrays passed by name are error-prone; echo results or use global naming conventions instead.
• Exit codes: reserve 0 success, 1 generic, 2 usage, 126 not executable, 127 not found, 128+N fatal signal. Document any script-specific codes.

Control Flow

if [[ -f $file && -r $file ]]; then ...; fi
[[ $name =~ ^[a-z][a-z0-9_]*$ ]] || die "invalid name"
case $1 in
  -v|--verbose) verbose=1 ;;
  -h|--help) usage; exit 0 ;;
  --) shift; break ;;
  *) die "unknown option: $1" ;;
esac
command -v jq >/dev/null || die "jq required"

• Prefer [[ ]] in bash (no word splitting, &&/||, =~, -v, lexicographic <). Use [ ] only for POSIX scripts; quote everything inside it.
• [[ a == b ]] is a pattern match on the right side unless quoted: [[ $x == "$y" ]].
• test -v var checks set-ness under set -u; ${var+x} is the portable equivalent.
• getopts handles short options only; long options need a while (($#)); do case ... loop. Never shift past $#.
• trap ... DEBUG and set -x are for debugging, not production; PS4='+${BASH_SOURCE}:${LINENO}: ' makes traces useful.

Portability: POSIX vs bash vs zsh

| Construct | POSIX sh | bash | zsh | |---|---|---|---| | [[ ]] | No | Yes | Yes (different regex) | | Arrays | No | Yes | Yes (1-indexed) | | local | Not specified (widely supported) | Yes | Yes | | set -o pipefail | No | Yes | Yes | | ${var//a/b} | No | Yes | Yes | | read -r -a arr | No | Yes | -A | | mapfile/readarray | No | 4.0+ | read -A | | printf -v | No | Yes | printf -v | | $RANDOM | No | Yes | Yes | | mktemp | Not specified but universal | Yes | Yes | | seq | Not POSIX (BSD differs) | GNU | BSD flavor |

• For POSIX scripts use #!/bin/sh, avoid arrays and [[ ]], use $(...) not backticks, and prefer case over getopts for long options.
• Test on the actual floor: dash -n script.sh catches bashisms fast; run in a container if the target uses BusyBox.
• command -v is the portable "is this installed" check; which is not reliable.

Testing and Static Analysis

# bats-core test
@test "usage exits 2" {
  run ./deploy.sh --bogus
  [ "$status" -eq 2 ]
  [[ $output == *"Usage:"* ]]
}

@test "dry run creates no files" {
  run ./deploy.sh --dry-run
  [ "$status" -eq 0 ]
  [ ! -e "$BATS_TEST_TMPDIR/out" ]
}

shellcheck -x -S style script.sh     # -x follows sourced files
shfmt -i 2 -ci -w script.sh          # deterministic formatting
bats --print-output-on-failure tests/ # run the suite

• shellcheck codes worth knowing: SC2086 (quote to prevent splitting), SC2046 (quote command substitution), SC2181 (check $? directly), SC2155 (declare/assign masks status), SC2164 (cd without || exit), SC2310/SC2311 (function masking -e).
• Disable rules narrowly with a reason: # shellcheck disable=SC2086 # intentional glob.
• bats-core uses $BATS_TEST_TMPDIR for isolated scratch space; setup_file runs once per file, setup before each test. Avoid asserting on timing.
• Test the failure paths: missing dependency, bad input, permission denied, interrupt. Unattended scripts fail at 3 a.m.; your tests should reach those branches.

Common Footguns

• CMD=$(command) > file redirects the outer assignment, not the command; use command > file.
• cd failure ignored, then rm -rf "$PWD"/* deletes the wrong tree. Always cd ... || die.
• Pipelines run in subshells: while read -r x; do n=$((n+1)); done < f works, but ... | while read loses n after the pipe.
• PIPESTATUS must be read immediately; any intervening command resets it.
• xargs without -0 breaks on spaces/newlines: pair with find -print0 and -0.
• find . -name '*.log' -delete races with new files; quote patterns so the shell does not expand them first.
• rm -rf -- "$dir/" with a typo'd variable (dir=/) is catastrophic: guard with ${dir:?}, use --, and prefer find -delete or targeted paths.
• Backticks nest badly and process escapes; always $(...).
• eval on untrusted input is code execution; ${!var} indirection is the safe form.
• Writing to a file in place (cmd > file) truncates before the command reads it; write to a temp file and mv.
• sudo in scripts requires a TTY by default; sudo -n fails fast, and BEGIN{...}.
• Locale-dependent sort/grep output changes under LC_ALL; set LC_ALL=C for deterministic parsing.
• Relying on $PATH in cron/systemd: set an explicit PATH= or use absolute binaries.

Anti-Patterns

• Parsing ls output; use globs, find, or shellcheck-clean loops.
• cat file | grep x; use grep x file (UUOC).
• for i in $(seq 1 10); use for ((i=1;i<=10;i++)) in bash or while for POSIX.
• set -e plus blanket || true, which nullifies strict mode.
• Long inline awk/sed scripts doing what a small Python script does legibly.
• Reading secrets from argv (visible in ps); read from env or file descriptor.
• chmod 777 to make a permission error disappear instead of fixing ownership.
• Ignoring shellcheck findings because "it works on my machine".

Checklist

• [ ] Shebang matches the constructs used; interpreter floor documented.
• [ ] set -Eeuo pipefail (bash) or deliberate POSIX equivalent.
• [ ] All expansions quoted; "$@" for positional parameters; -- before paths.
• [ ] Traps set for EXIT and INT/TERM; cleanup idempotent; 128+N exit codes.
• [ ] Temporaries via mktemp, cleaned on exit; no fixed /tmp filenames.
• [ ] Dependencies checked with command -v and a clear error.
• [ ] shellcheck and shfmt clean; bats tests cover failure paths.
• [ ] Logs go to stderr; machine-readable output goes to stdout.
• [ ] No secrets in argv, environment dumps, or set -x traces.
• [ ] Idempotent where plausible (safe to re-run after partial failure).

Related: Linux administration for scheduling and services, security hardening for privilege and secret handling.

Linux Administration

Administering mainstream Linux hosts: systemd services and timers, users and permissions, cgroups v2, namespaces, journald, packages, and scheduling.

Platform Landscape (2026)

| Family | Current stable lines | Notes | |---|---|---| | RHEL / Rocky / Alma | RHEL 9 and 10 lines | 10-year lifecycles; SELinux enforcing; dnf | | Fedora | Rolling yearly releases | Newest systemd/kernel; upstream validation | | Ubuntu LTS | 22.04, 24.04, 26.04 LTS line | apt; AppArmor default; snaps on server | | Debian | 12 (bookworm), 13 (trixie) lines | Long support; apt; minimal by default | | SUSE | SLES 15/16 lines, openSUSE Leap/Tumbleweed | zypper; YaST; btrfs default with snapshots | | Arch / derivatives | Rolling | pacman; expect manual intervention notices |

• Verify exact support windows and EOL dates upstream before planning upgrades; distro lifecycles differ by release. Never infer a patch date from general impressions.
• Modern systemd (v255 and later in 2026) provides deep sandboxing directives; consult systemd.exec(5) on the target host rather than assuming a directive exists.
• Kernel 6.x is standard; cgroups v2 unified hierarchy is the default on all current distros. Hybrid v1 hosts are legacy and should be migrated.

Boot and Service Inspection

systemctl --failed
systemctl status nginx --no-pager
systemd-analyze blame | head
systemd-analyze critical-chain sshd.service
journalctl -b -p err --since "1 hour ago"
hostnamectl; timedatectl

• systemd-analyze verify /etc/systemd/system/foo.service catches syntax and ordering errors before restarting.
• systemctl cat unit shows all drop-ins; systemctl show -p ... dumps effective values.
• Ordering (After=/Before=) is not dependency (Wants=/Requires=). Both are needed for the common "start after network but also pull it in" case.
• Requires= failure stops dependents; Wants= is best-effort. Use BindsTo= for lifetime coupling and PartOf= for restart propagation.

Writing systemd Units

# /etc/systemd/system/myapp.service
[Unit]
Description=My API
Documentation=https://example.invalid/docs
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
User=myapp
Group=myapp
Restart=on-failure
RestartSec=3
TimeoutStopSec=30
KillSignal=SIGTERM
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallFilter=@system-service
ReadWritePaths=/var/lib/myapp /var/log/myapp
LimitNOFILE=65536
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

| Type= | Use when | Signals readiness | |---|---|---| | simple | Default; process forks nothing | Immediately (false readiness) | | exec | Like simple but waits for exec to succeed | After exec | | notify | App calls sd_notify(READY=1) | Explicit; best for daemons | | forking | Legacy daemon double-forks | Parent exit | | oneshot | Task that exits; pair with RemainAfterExit= | On completion | | idle | Delay until other jobs finish | After queue drained |

• Never edit units in /usr/lib/systemd/system; place overrides in /etc/systemd/system/<unit>.d/10-override.conf via systemctl edit.
• daemon-reload after any unit change; restart (not reload) when ExecStart changes, because reload keeps the old process image.
• Sandbox errors surface as status=226/NAMESPACE or 203/EXEC; check journalctl -u unit and systemd-analyze verify before blaming the app.
• ProtectSystem=strict makes /usr, /boot, /etc read-only; list writable paths in ReadWritePaths=. This is the single highest-value hardening directive for services.

Timers vs cron

# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true
AccuracySec=1min

[Install]
WantedBy=timers.target

systemctl list-timers --all
systemctl start backup.timer
systemd-analyze calendar "Mon..Fri *-*-* 09:00:00"
journalctl -u backup.service --since today

| Need | Choice | Rationale | |---|---|---| | Scheduled job with logs and dependencies | systemd timer | journald, Persistent=, jitter, cgroup limits | | Simple periodic snippet, legacy host | cron | Universal; mail on output; watch PATH | | Run missed job after downtime | Timer Persistent=true | cron has no catch-up | | Per-user schedules | User timers (systemctl --user) or cron | User timers need lingering | | Anacron-style on minimal systems | cronie/dcron + anacron | No systemd present |

• Timers need a matching .service with the same basename; enabling the timer does not enable the service.
• OnCalendar uses systemd calendar syntax; always test with systemd-analyze calendar.
• cron environment is minimal: absolute paths, explicit PATH, no shell rc files. Set MAILTO= if output should be discarded deliberately.

Users, Groups, Permissions

useradd --create-home --shell /bin/bash --groups sudo deploy
usermod -aG docker deploy            # -a is mandatory; omission replaces groups
passwd -l olduser                     # lock password login
userdel --remove olduser
id deploy; groups deploy; getent passwd deploy

| Mechanism | Use for | Notes | |---|---|---| | POSIX modes (chmod) | Owner/group/other bits | umask 027 typical for servers | | ACLs (setfacl) | Per-user/per-dir grants | getfacl; survives most copies; check fs support | | Capabilities (setcap) | One narrow root privilege | Prefer over setuid binaries | | sudo rules | Delegated admin | /etc/sudoers.d/, visudo -c to validate | | systemd User= | Service identity | Cheaper than sudo for daemons | | SELinux/AppArmor | Mandatory access control | See hardening |

• Setuid binaries are a persistent risk; inventory with find / -xdev -perm -4000 -type f 2>/dev/null and remove unnecessary ones.
• Sticky bit on shared dirs (/tmp, upload dirs) prevents deletion of others' files.
• chmod g+s on a directory sets group inheritance for new files (setgid dir), which is often better than per-file group juggling.
• sudo environment: env_reset, secure_path, requiretty defaults vary; test rules with sudo -l -U user and validate syntax with visudo -c.
• Never give NOPASSWD: ALL broadly; scope commands and arguments. Argument globs are bypassable — prefer wrapper scripts with fixed paths.
• Home directory permissions matter: 700 for users with SSH keys; sshd refuses world-writable homes and authorized_keys.

cgroups v2

systemctl set-property myapp.service MemoryMax=2G CPUQuota=150%
systemctl set-property --runtime myapp.service IOWeight=100
cat /sys/fs/cgroup/system.slice/myapp.service/memory.current
cat /sys/fs/cgroup/system.slice/myapp.service/memory.events
cat /sys/fs/cgroup/system.slice/myapp.service/cpu.stat

| File | Meaning | Typical use | |---|---|---| | memory.max | Hard limit; OOM kills inside cgroup | Cap runaway services | | memory.high | Soft throttle threshold | Prefer over hard limit for latency | | memory.current / memory.peak | Usage now / high-water | Monitoring | | memory.events | oom, oom_kill, high counters | Detect pressure | | cpu.weight | Relative CPU share (1-10000) | Fair sharing between services | | cpu.max | "quota period" bandwidth cap | Hard CPU caps | | pids.max | Process/thread cap | Fork-bomb containment | | io.weight / io.max | I/O share / bandwidth ceilings | Noisy-neighbor control | | cgroup.pressure | PSI pressure per cgroup | Saturation signals |

• Every systemd service is in a cgroup; prefer systemctl set-property or unit directives (MemoryMax=, CPUQuota=, TasksMax=) over hand-editing the hierarchy.
• system.slice vs user.slice split resources by context; user-<uid>.slice caps a whole login session.
• v1 controllers (cpu, memory, blkio) are legacy; do not build new automation on /sys/fs/cgroup/<controller>/.
• Delegation: Delegate=yes lets a service manage its own subtree (used by container runtimes); needed for rootless containers to set limits.

Namespaces (operator view)

lsns                                  # list namespaces and owners
nsenter -t "$PID" -n ss -tulpn        # enter a process's network namespace
unshare --user --map-root-user --mount bash   # quick unprivileged playground
ip netns add lab && ip netns exec lab ip link

| Namespace | Isolates | Typical debugging use | |---|---|---| | mnt | Mount points | Inspect a container's view: nsenter -m | | pid | Process IDs | See container processes: nsenter -p | | net | Interfaces, routes, ports | nsenter -n ss -tulpn | | user | UID/GID mapping | Rootless containers | | uts | Hostname/domain | Container hostname surprises | | ipc | SysV/POSIX IPC | Shared memory issues | | cgroup | cgroup root view | Container cgroup visibility | | time | Clock offsets | Time namespace experiments |

• nsenter -t PID -a enters all namespaces; requires privilege unless the target is your own process. It is the primary tool for container debugging.
• Namespaces isolate visibility, cgroups limit consumption; neither is a security boundary by itself without user namespaces plus MAC and seccomp.

journald

journalctl -u nginx -f
journalctl -p warning -S "2026-01-01" -U "2026-01-02"
journalctl -k -b -1                    # kernel, previous boot
journalctl _PID=1234 -o json-pretty
journalctl --disk-usage
journalctl --vacuum-time=30d

• Persistent storage: create /var/log/journal/ (or set Storage=persistent in /etc/systemd/journald.conf); otherwise logs vanish at reboot.
• Bound growth with SystemMaxUse=, SystemMaxFileSize=, MaxRetentionSec=; default is 10% of the filesystem. Rotate with journalctl --rotate before vacuuming.
• journalctl -o cat strips metadata for piping; -o json is for tooling.
• Application logs to stdout/stderr are captured automatically for services. Do not double-write to files; use StandardOutput=journal.
• Remote shipping: systemd-journal-remote/-upload, or a collector reading the journal file format. Centralize before you need it.

Package Management

| Distro | Install | Update | Search owner | Verify | |---|---|---|---|---| | Debian/Ubuntu | apt install p | apt update && apt full-upgrade | dpkg -S file | apt-cache policy p | | RHEL/Fedora | dnf install p | dnf upgrade | rpm -qf file | dnf info p | | SUSE | zypper in p | zypper up | rpm -qf file | zypper info p | | Arch | pacman -S p | pacman -Syu | pacman -Qo file | pacman -Si p |

• Unattended security updates: unattended-upgrades (Debian/Ubuntu), dnf-automatic (RHEL family). Configure reboots separately and deliberately.
• Repository hygiene: pin third-party repos, prefer vendor packages, and avoid curl | sh installers on servers. Verify signatures.
• Container/universal packages: snaps and flatpaks are for desktop or vendor-pinned cases; check mount points and update channels before adopting on servers.
• Keep a manifest (dpkg -l, rpm -qa) in config management; do not hand-install packages that drift from the source of truth.
• Upgrades run in a maintenance window with console access; kernel and glibc updates need reboots and rollback plans.

Troubleshooting Quick Reference

dmesg -T --level=err,warn              # kernel ring buffer, human time
lsof -i :8080; ss -tulpn               # who owns a port
ps -eo pid,ppid,stat,etime,cmd --sort=-%mem | head
strace -f -p PID -tt -T -e trace=network  # syscall trail (use briefly)
df -h; df -i; lsblk -f                 # capacity and inodes
systemctl status; journalctl -b -p err # last stop for many issues

• OOM kills appear in dmesg as Out of memory: Killed process and in cgroup memory.events; correlate with performance.
• Disk full vs inodes full vs read-only remount (after I/O errors) are three different failures with the same symptom; check all three.
• A service that works manually but fails under systemd is usually environment (PATH, HOME), working directory, or sandboxing (ProtectSystem).

Anti-Patterns

• Editing vendor unit files in place instead of drop-ins.
• chmod -R 777 or chown -R nobody as a debugging step.
• Running services as root when a dedicated user and capabilities suffice.
• Disabling SELinux or AppArmor to make a service start.
• cron jobs without absolute paths, error handling, or output redirection.
• Hand-editing /etc/passwd//etc/shadow instead of useradd/usermod/passwd.
• Ignoring systemctl --failed after boot.
• Unbounded journald or package caches filling the root filesystem.
• Kernel upgrades applied without a reboot plan or console access.

Checklist

• [ ] Host is on a supported distro release; EOL tracked.
• [ ] Unattended security updates enabled, reboot policy decided.
• [ ] Services run as dedicated users with sandboxing directives where practical.
• [ ] Units verified (systemd-analyze verify) and drop-in overrides recorded.
• [ ] Timers preferred over cron; Persistent=true for catch-up jobs.
• [ ] cgroup limits (MemoryMax, CPUQuota, TasksMax) set on risky services.
• [ ] journald persistent and size-bounded; critical logs shipped off-host.
• [ ] NTP synchronized; timezone and hostname correct.
• [ ] Setuid inventory reviewed; sudo rules minimal and validated.
• [ ] Root filesystem capacity and inode headroom monitored.

Related: storage, performance, containers, hardening.

macOS Administration

Administering and automating Macs: launchd, Homebrew, SIP/Gatekeeper, defaults, configuration profiles, TCC privacy, and APFS quirks.

Platform Landscape (2026)

| Topic | State | Notes | |---|---|---| | Current release line | macOS 26 (2025 naming scheme) | Verify the exact current version and support matrix upstream | | Still-supported older lines | Typically current and two prior | Apple publishes security update scope per release | | Hardware | Apple silicon (M-series) is the default | Intel Macs only on older supported releases | | Scripting shell | zsh 5.x at /bin/zsh; bash 3.2 at /bin/bash | Never assume bash 4+ features in system scripts | | Package manager | Homebrew on Apple silicon at /opt/homebrew | Intel path /usr/local; never hardcode both | | Init/scheduler | launchd (PID 1) | cron exists but is deprecated in practice |

• Apple silicon changed defaults: Homebrew prefix, some paths, and the behavior of virtualization; do not port Intel-era scripts blindly.
• System integrity is strict: /System is a read-only signed volume, SIP blocks most modifications even as root, and third-party kernel extensions are effectively legacy.
• MDM (Jamf, Kandji, Mosyle, Intune) is the supported fleet automation path; scripting alone does not scale or survive OS upgrades reliably.

launchd: Jobs and Services

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key><string>com.example.sync</string>
  <key>ProgramArguments</key>
  <array><string>/usr/local/bin/sync</string><string>--once</string></array>
  <key>StartInterval</key><integer>3600</integer>
  <key>RunAtLoad</key><true/>
  <key>StandardOutPath</key><string>/var/log/example-sync.log</string>
  <key>StandardErrorPath</key><string>/var/log/example-sync.err</string>
  <key>EnvironmentVariables</key>
  <dict><key>PATH</key><string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string></dict>
</dict>
</plist>

launchctl bootstrap system /Library/LaunchDaemons/com.example.sync.plist
launchctl kickstart -k system/com.example.sync
launchctl print system/com.example.sync
launchctl bootout system/com.example.sync
plutil -lint /Library/LaunchDaemons/com.example.sync.plist

| Location | Type | Runs as | Loaded | |---|---|---|---| | /Library/LaunchDaemons | Daemon | root (or UserName=) | Boot, system context | | /Library/LaunchAgents | Agent | logged-in user | User login, all users | | ~/Library/LaunchAgents | Agent | owner | That user only | | /System/Library/... | Apple | Apple | Never modify |

• launchctl load/unload is legacy; use bootstrap/bootout with a domain (system, gui/<uid>, user/<uid>).
• KeepAlive (with optional conditions) restarts on exit; ThrottleInterval (default 10s) prevents crash loops. StartCalendarInterval mimics cron.
• launchd jobs get a minimal environment: absolute paths, no .zshrc, and PATH must include Homebrew's bin explicitly. This is the top cause of "works in terminal".
• StartInterval and StartCalendarInterval jobs that miss a wake window do not run; the Mac must be awake. pmset schedule and caffeinate interact with this.
• Logs stdout/stderr to files (launchd does not capture to unified logging for job output). Use log show --last 1h --predicate 'process == "sync"' for os_log output.
• Disable a job without deleting it: launchctl disable system/com.example.sync.
• SIP protects /System; launchd daemons belong in /Library/LaunchDaemons, never inside app bundles you modify.

Homebrew

brew --prefix                     # /opt/homebrew (arm64), /usr/local (x86_64)
brew install jq gh                # formulae
brew install --cask docker        # GUI apps
brew bundle dump --file Brewfile
brew bundle install --file Brewfile
brew update && brew upgrade && brew cleanup -s
brew services list                # launchd-backed services
brew doctor
brew pin python@3.13

• Never run brew as root or via sudo; it refuses and breaks ownership. Use brew services (per-user) rather than system daemons for Homebrew software.
• Brewfile is the reproducibility unit for workstations: commit it, install with brew bundle, and review diffs like any dependency file.
• Homebrew is not a patch-management or MDM replacement; critical software should come from managed channels.
• Rosetta runs x86_64 formulae in separate prefixes when needed; prefer native arm64 bottles and mix only when a dependency requires it.
• brew bundle cleanup --force removes packages not in the Brewfile; run with care on machines with manual installs.

SIP, Gatekeeper, Notarization

csrutil status                       # System Integrity Protection
spctl --status                       # Gatekeeper assessment
xattr -l /Applications/Some.app      # look for com.apple.quarantine
spctl -a -vv --type execute Some.app # why was it blocked
codesign -dv --verbose=4 Some.app    # signature and team ID
codesign --verify --deep --strict Some.app

| Control | Blocks | Admin bypass | |---|---|---| | SIP | Modification of system paths, code injection | csrutil disable from Recovery only; avoid | | Gatekeeper | Unsigned/unnotarized apps | spctl --master-disable weakens the whole machine | | Notarization | Distribution of un-notarized binaries | Developer staple via notarytool | | Quarantine attribute | First-run of downloaded files | xattr -d com.apple.quarantine per-file |

• Production Apple software must be signed with a Developer ID and notarized; verify the full chain with codesign --verify --deep --strict and spctl -a -vv.
• Removing quarantine is a per-file workaround, not a deployment strategy; do not script it wholesale.
• MDM can allow specific apps and system extensions; user-approved kernel/system extension flows have largely replaced kexts (systemextensionsctl list).
• csrutil authenticated-root and sealed system volumes mean even root cannot edit /System; install into /usr/local or /opt instead.

defaults, TCC, and Privacy

defaults read com.apple.dock
defaults write com.apple.dock autohide -bool true && killall Dock
defaults read ~/Library/Preferences/com.example.app.plist
defaults delete com.example.app KeyName

• defaults writes preferences; some daemons cache values, so restart the app or killall the relevant process. Per-host managed preferences override user values.
• defaults is not a supported configuration mechanism for every setting; Apple may change keys between releases. Prefer MDM profiles for anything fleet-wide.
• TCC (Transparency, Consent, Control) gates camera, microphone, screen recording, accessibility, and full disk access. Databases are protected by SIP; do not attempt to edit them directly.
• tccutil reset <service> <bundle-id> resets prompts for testing. End users or MDM must grant consent; scripts cannot click the dialogs.
• Full Disk Access must be granted to backup and security tools explicitly; this is a common automation blocker.
• Screen recording and accessibility permissions are required for remote-control and UI-testing tools; document them in onboarding.

Configuration Profiles and MDM Basics

profiles list                        # installed profiles (may need root)
profiles show -type configuration
profiles -I -F profile.mobileconfig  # install manually (dev only)

• A .mobileconfig profile is a signed plist with PayloadType entries (com.apple.security.firewall, com.apple.MCX, com.apple.SoftwareUpdate, etc.).
• Profiles can be user-scoped or device-scoped; device profiles require MDM or admin.
• Declarative device management (DDM) is the modern MDM model: the device reports status and applies declarations, rather than the server pushing profiles blindly. Verify current MDM support upstream.
• Manual profiles install is for testing only: unsigned or unreviewed profiles are a security risk, and user-installed profiles can be removed by the user unless the device is supervised.
• FileVault, firewall, software update deferral, and password policy are the common baseline payloads. Keep a written baseline per device class.
• sudo fdesetup status confirms FileVault; escrow recovery keys through MDM for fleet devices.

Filesystem Quirks (APFS)

diskutil list; diskutil apfs list
diskutil info / | grep -E 'File System|Volume Name|Case'
tmutil listlocalsnapshots /
tmutil localsnapshot
tmutil listbackups; tmutil status
mount | grep -E 'apfs|hfs'

• APFS is the default and mandatory for boot volumes; HFS+ only remains on some external/legacy media.
• Boot volumes are a volume group: a read-only signed System volume and a writable Data volume joined by firmlinks. Paths like /Users, /Applications, and /private/var live on the Data volume.
• Default formatting is case-insensitive (case-preserving); code and archives that assume case sensitivity must be tested. Some build systems require a case-sensitive volume.
• APFS snapshots are instantaneous and cheap, but Time Machine local snapshots accumulate and consume space until purged; tmutil thinlocalsnapshots frees them.
• df output is misleading on APFS: containers share free space, and purgeable space is not reported as available until reclaimed. Use diskutil apfs list and df -h together.
• Time Machine to a network destination uses sparse bundles; verify backups with tmutil verifychecksums and periodic restore drills.
• Extended attributes and resource forks survive on APFS but are lost on FAT/exFAT media; zip archives need --keepResourceFork equivalents or ditto to preserve metadata for app bundles.

Security Baseline

• Enable FileVault, the application firewall (/usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate), and automatic security updates where policy allows.
• Track OS version against Apple's security release notes; Macs fall behind silently.
• Audit login items and background items: sfltool dumpbtm (privileged), System Settings > General > Login Items, and launchctl print for stray agents.
• Secure Enclave backs FileVault keys and Touch ID; do not attempt to script around it.
• XProtect and Gatekeeper are not a complete EDR; pair them with MDM and endpoint tooling for fleets.

Anti-Patterns

• sudo with Homebrew, or installing Homebrew as root.
• Editing /System or TCC databases directly.
• Writing daemons into ~/Library/LaunchAgents for system-wide services.
• Assuming cron behavior; macOS may sleep through scheduled cron/timer runs.
• Hardcoding /usr/local or /opt/homebrew without detecting brew --prefix.
• Parsing system_profiler output for structured data when diskutil -plist or plutil provides stable formats.
• Disabling SIP/Gatekeeper to make an unsigned tool run on a managed fleet.
• Ignoring local Time Machine snapshots when diagnosing "disk full".

Checklist

• [ ] Fleet devices enrolled in MDM; baseline profiles documented and versioned.
• [ ] FileVault on with escrowed recovery key; firewall enabled.
• [ ] Software update policy explicit; deferrals deliberate, not accidental.
• [ ] Homebrew prefix discovered dynamically; Brewfile committed for workstations.
• [ ] LaunchDaemons vs LaunchAgents chosen correctly; labels reverse-DNS.
• [ ] Job PATH and filesystem paths absolute; outputs logged to files.
• [ ] Scripts pass plutil -lint for plists and shellcheck for shell.
• [ ] Full Disk Access and TCC-sensitive tooling granted deliberately.
• [ ] Time Machine or equivalent verified with a restore test.
• [ ] No reliance on deprecated load/unload or unsupported defaults keys.

Related: shell scripting for script hygiene, security hardening for cross-platform controls.

Windows Administration

Automating and operating Windows hosts: PowerShell 7, winget, WSL2, Task Scheduler, services, Event Log, and basic Active Directory.

Platform Landscape (2026)

| Product | State | Notes | |---|---|---| | Windows 11 | 24H2/25H2-era annual releases | Current consumer/business client line | | Windows 10 | End of standard support reached (Oct 2025) | ESU or upgrade required; verify ESU terms upstream | | Windows Server | 2019, 2022, 2025 lines | LTSC; verify mainstream support windows upstream | | PowerShell | 7.4 LTS line and later 7.x stable lines | Windows PowerShell 5.1 remains inbox; 7.x is the automation target | | WSL | WSL2 with a Microsoft kernel | Distro comes from the Store; systemd supported | | Package manager | winget | Chocolatey/Scoop remain niche but useful |

• Windows 10 EOL drives most 2026 migration work; inventory before planning.
• Prefer PowerShell 7 for new automation: cross-platform, faster, better ForEach-Object -Parallel, native SSH remoting. Keep 5.1-compatible scripts only where required by inbox-only servers.
• Update strategy: Windows Update for Business, WSUS, or a patch management tool; monthly cumulative updates plus out-of-band advisories. Track build numbers, not just marketing versions.

PowerShell 7 Patterns

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$PSNativeCommandUseErrorActionPreference = $true   # 7.3+: native exit codes become errors

function Get-DiskReport {
    [CmdletBinding()]
    param(
        [Parameter(Mandatory)][string[]]$ComputerName,
        [ValidateRange(1, 100)][int]$Threshold = 10
    )
    process {
        foreach ($name in $ComputerName) {
            Get-CimInstance -ClassName Win32_LogicalDisk -ComputerName $name |
                Where-Object { $_.DriveType -eq 3 } |
                Select-Object @{n = 'Host'; e = { $name } }, DeviceID,
                    @{n = 'FreeGB'; e = { [math]::Round($_.FreeSpace / 1GB, 1) } },
                    @{n = 'TotalGB'; e = { [math]::Round($_.Size / 1GB, 1) } }
        }
    }
}

| Pattern | Do | Avoid | |---|---|---| | Error handling | -ErrorAction Stop, try/catch, $ErrorActionPreference | $? checks after every call | | Pipeline | Emit objects; filter with Where-Object | String parsing with Select-String | | Output | Return objects; format at the edge | Write-Host for data | | Parameters | [CmdletBinding()], typed, validated | $args and positional guessing | | Idempotence | Test-Path, Get-*/Set-*, desired-state checks | Blind New-* on every run | | Credentials | PSCredential, managed identity, Windows Hello | Plaintext or argv passwords | | Remoting | Invoke-Command over WinRM/SSH | PsExec for everything | | Modules | Version-pinned modules, Install-Module -Scope AllUsers | Unpinned gallery installs |

• $PSNativeCommandUseErrorActionPreference (7.3+) makes non-zero exit codes from native tools throw under Stop; without it, native failures pass silently.
• Set-StrictMode -Version Latest catches uninitialized variables and bad property access; add it to every non-trivial script.
• Get-CimInstance replaces Get-WmiObject (removed in 7). CIM cmdlets support -ComputerName and DCOM/WinRM sessions.
• Approved verbs only (Get-Verb); -WhatIf/-Confirm via SupportsShouldProcess.
• Classes and using namespace work in 7 but for reusable code prefer modules with a .psd1 manifest and semantic versioning.
• Test with Pester 5: Describe/Context/It, Should -Be, Mock for external calls; run in CI with Invoke-Pester -CI.
• Script signing: production scripts are signed (Authenticode) and execution policy is RemoteSigned or AllSigned; execution policy is not a security boundary, AppLocker or WDAC is.
• $PSStyle (7.2+) controls ANSI colors; disable formatting for logs.

winget

winget search --name git
winget install --id Git.Git -e --source winget --accept-package-agreements --accept-source-agreements
winget list --upgrade-available
winget upgrade --all --include-unknown
winget export -o packages.json
winget import -i packages.json --accept-package-agreements
winget configure -f config.dsc.yaml     # WinGet Configuration (DSC-based)

• --id plus -e (exact) avoids fuzzy-match installs of the wrong package.
• Machine-wide installs require an elevated shell; per-user installs do not. Decide per package and document it.
• winget is not installed on older Windows Server builds by default; bootstrap from the App Installer package or use Chocolatey where winget is unavailable.
• winget configure applies DSC v3-style YAML documents; treat the YAML as the source of truth for developer/workstation baselines.
• Private repositories and internal package sources are supported; prefer them for controlled software distribution.
• Chocolatey (choco install, choco upgrade all) remains useful for servers and automation-heavy environments; pin versions and review community package scripts.

WSL2

# %UserProfile%\.wslconfig  (global, per user)
[wsl2]
memory=8GB
processors=4
swap=2GB
networkingMode=mirrored
firewall=true
[experimental]
autoMemoryReclaim=gradual
sparseVhd=true

# /etc/wsl.conf  (inside the distro)
[boot]
systemd=true
[interop]
appendWindowsPath=false
[automount]
options="metadata,umask=22,fmask=11"

wsl --install -d Ubuntu
wsl -l -v
wsl --shutdown
wsl --export Ubuntu D:\backups\ubuntu.tar
wsl --import Ubuntu2 D:\wsl\ubuntu2 D:\backups\ubuntu.tar --version 2

• WSL2 is a real Linux kernel in a lightweight VM: systemd, cgroups v2, and containers work inside it. WSL1 is legacy; avoid for new work.
• networkingMode=mirrored (Windows 11 22H2+) gives localhost and LAN parity with the host; classic NAT mode requires localhostForwarding and separate firewall rules.
• WSL distro files live in a VHDX; wsl --shutdown before backup/copy, then export or copy the VHDX. Backups are the user's responsibility.
• Cross-filesystem performance: operating on files under /mnt/c is slow. Keep project files in the Linux filesystem (~/) and reach them from Windows via \\wsl$.
• appendWindowsPath=false prevents Windows PATH pollution in Linux shells; call powershell.exe/cmd.exe explicitly for interoperability.
• Time drift after host sleep is corrected by wsl --shutdown/restart; NTP inside the distro may not fix the VM clock.
• WSL is supported on Windows 10 21H2+ and Windows 11/Server 2022+; verify host build before standardizing.
• GPU (CUDA/WSLg) and USB (usbipd-win) passthrough are supported but version-sensitive; verify upstream prerequisites.

Task Scheduler

$action = New-ScheduledTaskAction -Execute 'pwsh.exe' `
    -Argument '-NoProfile -File C:\ops\sync.ps1'
$trigger = New-ScheduledTaskTrigger -Daily -At 3am
$principal = New-ScheduledTaskPrincipal -UserId 'SYSTEM' `
    -LogonType ServiceAccount -RunLevel Highest
$settings = New-ScheduledTaskSettingsSet -StartWhenAvailable `
    -DontStopOnIdleEnd -ExecutionTimeLimit (New-TimeSpan -Hours 2) `
    -MultipleInstances IgnoreNew
Register-ScheduledTask -TaskName 'Ops Sync' -Action $action `
    -Trigger $trigger -Principal $principal -Settings $settings -Force

Get-ScheduledTask -TaskName 'Ops Sync' | Get-ScheduledTaskInfo
Start-ScheduledTask -TaskName 'Ops Sync'
Disable-ScheduledTask -TaskName 'Ops Sync'

• schtasks /query /tn "Ops Sync" /xml is the legacy path; the *-ScheduledTask cmdlets are preferred and map 1:1 to the XML.
• -StartWhenAvailable is the catch-up equivalent of systemd Persistent=true.
• Principal matters: SYSTEM for machine-wide work, a service account for network access (SYSTEM lacks network credentials on other hosts), interactive users for GUI.
• MultipleInstances controls overlap: IgnoreNew prevents pileups, Queue queues.
• Actions run with the working directory System32 by default; set -WorkingDirectory.
• Task history lives under Microsoft-Windows-TaskScheduler/Operational; enable it explicitly, it is off by default.
• Store scripts on a path the principal can read; SYSTEM cannot see user-mapped drives.

Services

Get-Service | Where-Object Status -eq 'Running'
New-Service -Name 'MyApp' -BinaryPathName '"C:\Program Files\MyApp\app.exe" --service' `
    -DisplayName 'My App' -StartupType Automatic -Description 'API worker'
sc.exe failure MyApp reset= 86400 actions= restart/5000/restart/10000/none/0
Restart-Service MyApp
Get-CimInstance Win32_Service -Filter "Name='MyApp'" | Select State, StartMode, PathName

• Native Windows services must implement the Service Control Handler or use a wrapper (WinSW, NSSM) for ordinary executables. A plain console app will time out at startup.
• Recovery policy (sc.exe failure) is the restart-on-failure equivalent; set it explicitly, the default does nothing.
• Service accounts: prefer virtual accounts (NT SERVICE\MyApp, gMSA) over LocalSystem or shared passwords. Grant only required privileges and file rights.
• Delayed start avoids boot-time contention: sc.exe config MyApp start= delayed-auto.
• Service start timeout (default 30s) is configurable per service; fix slow startup rather than extending blindly.

Event Log

Get-WinEvent -LogName System -MaxEvents 50 |
    Where-Object LevelDisplayName -in 'Error', 'Critical' |
    Format-Table TimeCreated, Id, ProviderName, Message -Wrap

$filter = @{ LogName = 'Application'; Level = 1, 2
             StartTime = (Get-Date).AddDays(-1) }
Get-WinEvent -FilterHashtable $filter |
    Group-Object Id, ProviderName | Sort-Object Count -Descending

wevtutil qe Security /q:"*[System[(EventID=4625)]]" /f:text /c:20
wevtutil epl System C:\diag\system.evtx        # export for offline analysis

• Get-WinEvent -FilterHashtable is server-side filtering (fast); Where-Object is client-side (slow) — filter first, format later.
• Event ID landmarks: 4624/4625 logon success/failure, 4672 special privileges, 4688 process creation (requires audit policy), 7045 service installed, 6008 unexpected shutdown, 1074 clean shutdown/restart reason.
• Audit policy must be enabled for Security events (4688, 4697, etc.): auditpol /get /category:*; set via GPO or auditpol /set.
• Log sizes are capped; increase with wevtutil sl System /ms:104857600 for busy hosts.
• Centralize with Windows Event Forwarding (WEF) or an agent; local logs are lost on disk failure and rolled over quickly.

Active Directory Basics

Import-Module ActiveDirectory
Get-ADUser -Filter "Enabled -eq 'True'" -Properties LastLogonDate |
    Sort-Object LastLogonDate | Select-Object -First 20 Name, LastLogonDate
Get-ADGroupMember 'Domain Admins' | Select Name, SamAccountName
Get-ADComputer -Filter 'OperatingSystem -like "*Server*"' -Properties OperatingSystem |
    Group-Object OperatingSystem
Search-ADAccount -AccountDisabled -UsersOnly | Select Name, SamAccountName
gpresult /h gpo-report.html; gpupdate /force

• RSAT (Install-WindowsFeature RSAT-AD-PowerShell or the Windows capabilities) is required; install on admin workstations, not servers, where possible.
• LDAP queries against a Domain Controller for read-heavy tasks; the AD module is a wrapper. Use paging and filters, never enumerate the whole domain.
• Kerberos: SPNs must be unique; setspn -L account audits registration. Clock skew beyond 5 minutes breaks authentication; ensure W32Time/PDC synchronization.
• Group Policy preferences can embed credentials (cpassword) — audit and eliminate. LAPS for local admin passwords, tiered administrative model for privileged accounts.
• Entra ID (hybrid/cloud) increasingly holds identity; sync via Entra Connect, and treat conditional access as the control plane for cloud apps.
• Recycle Bin, fine-grained password policies, and protected users group are baseline hygiene items to verify, not assume.

Anti-Patterns

• Using Windows PowerShell 5.1 idioms in 7 (Get-WmiObject, Measure-Command ||, etc.) without a compatibility reason.
• Write-Host for data or logging; it bypasses the pipeline and cannot be suppressed.
• Invoke-Expression on strings; it is arbitrary code execution.
• Storing credentials in scripts or Scheduled Task arguments.
• Using SYSTEM for tasks that need network resource access.
• Assuming winget is present on servers, or that Store apps install in audit/SYSTEM contexts.
• Editing the registry with reg add when a cmdlet, GPO, or DSC resource exists.
• Treating execution policy as a security boundary.
• Editing files under C:\Windows or using takeown/icacls to force permissions.

Checklist

• [ ] Windows 10 inventory complete; ESU/upgrade path decided.
• [ ] PowerShell 7 deployed; modules version-pinned and signed scripts where required.
• [ ] winget/Chocolatey baseline captured as code (YAML/Brewfile-equivalent).
• [ ] WSL2 configured with .wslconfig and /etc/wsl.conf; backups exported.
• [ ] Scheduled tasks use explicit principals, settings, and start-when-available.
• [ ] Services have recovery policy and least-privilege accounts.
• [ ] Event log sizes raised; Security auditing enabled per baseline; logs forwarded.
• [ ] AD privileged groups reviewed; LAPS and tiered admin understood.
• [ ] Patch cadence defined (WUfB/WSUS) with reboot windows.
• [ ] Backups for host state, not just data; restore tested.

Related: security hardening for account and patch baselines, storage for volume and backup planning.

Networking

Diagnosing and configuring network paths on Linux/macOS/Windows hosts: DNS, TCP and TLS lifecycles, packet and socket tools, nftables, tunnels/VPN, and proxy debugging.

Diagnostic Model

Work bottom-up and stop at the first broken layer. Each layer has a cheap test.

| Layer | Question | Command | Common failure | |---|---|---|---| | Link | Is the interface up/carrier? | ip -br link, ethtool eth0 | Cable, VLAN, driver, MTU | | IP | Do we have a valid address? | ip -br addr | DHCP, duplicate IP, wrong subnet | | Route | Does a path exist? | ip route, ip rule | Missing default, policy routing | | Neighbor | Is the gateway reachable? | ip neigh, arping | ARP/NDP, proxy ARP | | DNS | Does the name resolve? | dig +short name, resolvectl query | Wrong resolver, NXDOMAIN, search domain | | TCP | Does the port answer? | nc -vz host port, ss -tulpn | Firewall drop, no listener, backlog | | TLS | Does the handshake validate? | openssl s_client -connect | Expired/wrong chain, SNI, clock | | App | Does the request succeed? | curl -sv -o /dev/null | Auth, headers, timeouts, proxy |

• Drops are silent; rejects are loud. A firewall DROP produces timeouts, REJECT produces connection-refused or ICMP errors. Use that to distinguish causes.
• Always compare from the same vantage point: localhost vs LAN vs internet behave differently because of firewall rules, NAT, and proxies.
• Capture before changing: tcpdump -i any -nn -c 200 on both ends when possible.

DNS

dig example.com A +short
dig example.com MX TXT
dig @1.1.1.1 example.com A +norecurse
dig +trace example.com
dig -x 203.0.113.10 +short
dig +tcp example.com                 # force TCP (truncation debugging)
drill example.com A                   # macOS/BSD alternative
resolvectl status; resolvectl query example.com

| Record | Purpose | Gotcha | |---|---|---| | A/AAAA | Host to IPv4/IPv6 | Dual-stack: clients may prefer IPv6 and fail | | CNAME | Alias | Cannot coexist with other records at the apex | | MX | Mail routing | Priority order, must point to A/AAAA | | TXT | Verification, SPF/DKIM/DMARC | 255-char string chunks; aggregate limits | | SRV | Service discovery | Weight/priority; underscores in names | | NS/SOA | Delegation and zone metadata | Negative TTL from SOA affects caching | | CAA | Which CAs may issue certs | Missing CAA blocks issuance |

• System resolution path differs: glibc nsswitch.conf (files, then DNS via resolv.conf), systemd-resolved (resolvectl, stub at 127.0.0.53), macOS scutil --dns (per-interface resolvers + search domains), Windows ipconfig /all and Get-DnsClientServerAddress.
• dig bypasses the system resolver and talks directly to the server listed; when dig @1.1.1.1 works but getent hosts fails, the local resolver configuration is at fault, not the zone.
• TTL discipline: lower TTLs before planned migrations, then restore. Clients ignore TTLs you wish they would respect; plan for the full propagation window.
• Search domains and ndots cause surprising names to resolve; trailing dot in dig and curl names (e.g., example.com.) avoids search-list interference.
• DoT/DoH encrypt queries but not the SNI/IP metadata; enterprise resolvers may be mandatory. Verify which resolver the OS actually uses before blaming DNS.
• Split-horizon zones return different answers internally vs externally; test from both sides.
• Negative answers are cached; NXDOMAIN lingering after a fix is usually TTL.

TCP Lifecycle

ss -tan state established '( sport = :443 )'
ss -s                                  # summary by state
ss -tulpn                              # listeners with process
sysctl net.ipv4.tcp_congestion_control  # bbr on modern kernels
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog

| State | Meaning | Concern | |---|---|---| | LISTEN | Accepting connections | Backlog overflow drops SYNs | | SYN-SENT | Client handshake in flight | Packets filtered or host down | | SYN-RECV | Server saw SYN, sent SYN-ACK | SYN flood or slow app | | ESTABLISHED | Data transfer | Socket counts, file descriptors | | TIME-WAIT | Normal close by local end | Port exhaustion in extreme cases | | CLOSE-WAIT | Peer closed, app did not | Application leak, FD exhaustion | | FIN-WAIT-2 | Half-closed | Peer not closing; often benign |

• Three-way handshake: SYN → SYN-ACK → ACK. tcpdump showing repeated SYN with no SYN-ACK means the server side is dropping (firewall, no listener, or full backlog).
• Congestion control: CUBIC is the common default, BBR is widely deployed on modern kernels; verify with sysctl net.ipv4.tcp_available_congestion_control. Do not switch based on blog posts; benchmark on the actual path.
• RTT and loss drive throughput far more than buffer tuning. Measure first with mtr or ping -c series before touching tcp_rmem/tcp_wmem.
• MTU and PMTUD: tunnels and VPNs shrink the path MTU; black-holed ICMP breaks PMTUD and produces hangs on large packets. Test with ping -M do -s 1472 host (1400 payload for tunneled paths).
• Connection limits: ss -s counts sockets; process FD limits matter more than kernel tables in practice. See performance for ulimits.

TLS Lifecycle

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null
openssl s_client -connect example.com:443 -tls1_3 -brief
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates -subject -issuer
openssl x509 -in cert.pem -noout -text | grep -A1 'Subject Alternative Name'
curl -sv https://example.com -o /dev/null

• Handshake order: TCP connect → ClientHello (SNI, ALPN, cipher list) → ServerHello and certificate chain → client validation → key exchange → Finished. TLS 1.3 collapses this to one round trip; session resumption (PSK) removes another.
• Validate the full chain from leaf to root: mismatched intermediate order, missing intermediates, or an untrusted root break some clients and not others.
• Certificate checks: hostname must be in SAN (CN fallback is dead), validity window must include now, key usage/EKU must permit server auth. openssl verify -CAfile is the offline check.
• SNI selects the certificate; hitting a shared IP by IP address often yields the wrong vhost cert, a common false alarm.
• Clock skew invalidates certificates; check timedatectl/w32tm when TLS fails mysteriously.
• ACME automation (certbot, lego, acme.sh) with HTTP-01 or DNS-01; DNS-01 enables wildcards. Renew with jitter and monitor expiry at 30/14/7 days.
• mTLS: server requests client cert; verify the client chain and CRL/OCSP path. Treat client cert expiry as an outage.
• TLS 1.0/1.1 are deprecated; target 1.2+ with 1.3 preferred. Old clients must be a documented exception.

Packet and Socket Tools

# curl: timings and virtual host resolution
curl -sS -o /dev/null -w 'dns=%{time_namelookup} conn=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' https://example.com
curl -sv --resolve example.com:443:203.0.113.10 https://example.com/
curl -x http://proxy:3128 -v https://example.com/
ssh -v host; ssh -vvv host          # two/three levels of connection debugging

# tcpdump: always bound the capture
tcpdump -i any -nn -c 100 port 443
tcpdump -i eth0 -nn 'host 203.0.113.10 and (port 80 or port 443)'
tcpdump -i any -nn -A 'tcp port 80 and (((ip[2:2]-((ip[0]&0xf)<<2))-((tcp[12]&0xf0)>>2)) != 0)'
tcpdump -i any -nn -w /tmp/cap.pcap 'port 53'   # analyze in Wireshark later

# path and reachability
mtr -rwzbc 50 example.com
traceroute -T -p 443 example.com
nmap -Pn -p 22,80,443 203.0.113.0/24

• tcpdump -nn avoids DNS/service lookups; without -c or a filter you will flood the terminal. Write to pcap when analysis is non-trivial.
• ss -i shows congestion window and RTT per connection; ss -K kills sockets.
• curl -w timing breakdown localizes slowness to DNS, connect, TLS, or application.
• Get-NetTCPConnection and Test-NetConnection -Port are the Windows equivalents; pktmon captures packets on modern Windows.
• On macOS, nettop, networkQuality, and scutil --dns fill gaps; pfctl is the firewall.

nftables

nft list ruleset
nft -f /etc/nftables/edge.nft

#!/usr/sbin/nft -f
flush ruleset

table inet filter {
  set ssh_burst {
    type ipv4_addr
    flags dynamic, timeout
    timeout 1m
  }

  chain input {
    type filter hook input priority filter; policy drop;
    ct state established,related accept
    ct state invalid drop
    iif lo accept

    ip protocol icmp accept
    ip6 nexthdr ipv6-icmp accept

    tcp dport 22 ct state new \
      meter ssh_burst { ip saddr limit rate over 4/minute } drop
    tcp dport 22 accept
    tcp dport { 80, 443 } accept
  }

  chain forward {
    type filter hook forward priority filter; policy drop;
  }

  chain output {
    type filter hook output priority filter; policy accept;
  }
}

• nftables replaces iptables/ip6tables/arptables/ebtables; iptables-nft is a compatibility shim. Do not mix direct nft rules with iptables shim rules.
• Policy drop with ct state established,related accept first is the standard stateful baseline; the conntrack table must be sized for the workload.
• Sets and maps scale to large address lists; flags dynamic, timeout enables rate-limiting meters without extra tooling.
• Persist atomically: distro unit loads /etc/nftables.conf (Debian/Arch) or /etc/sysconfig/nftables.conf (RHEL family) with nft -f, or use nft list ruleset > file after live edits and review the diff.
• firewalld (RHEL) and ufw (Ubuntu) are frontends; use one layer consistently, do not manage the same host with both nft and a frontend.
• NAT for routers: chain postrouting { type nat hook postrouting priority srcnat; oifname "eth0" masquerade; }; container runtimes manage their own NAT tables — keep your rules separate.

Tunnels and VPN

# /etc/wireguard/wg0.conf
[Interface]
Address = 10.8.0.1/24
ListenPort = 51820
PrivateKey = <server-private-key>

[Peer]
PublicKey = <client-public-key>
AllowedIPs = 10.8.0.2/32
PersistentKeepalive = 25

| Option | Use when | Notes | |---|---|---| | WireGuard | Site-to-site, remote access | In kernel; minimal config; UDP only | | OpenVPN | Legacy peers, TCP/443 fallback | Userspace, heavier, mature ecosystem | | SSH -L/-R | Ad-hoc forwarding | No persistent routing; great for debugging | | SSH -D | SOCKS proxy | Combine with browser/app proxy config | | Tailscale/Headscale | Mesh, identity-based access | Control plane dependency; ACLs matter | | IPsec | Standards-mandated site links | Complex; strong interoperability |

wg show; wg-quick up wg0
ssh -N -L 8080:internal:80 jump-host       # local forward
ssh -N -R 9000:localhost:9000 public-host  # remote forward
ssh -D 1080 -N jump-host                    # SOCKS5 proxy

• AllowedIPs doubles as routing: on the client, 0.0.0.0/0 routes all traffic (full tunnel); narrow ranges are split tunnel. Be explicit.
• WireGuard has no built-in user auth; distribute keys per device, rotate, and revoke by removing peers. Keep private keys in files with mode 600.
• MTU: WireGuard overhead is ~60 bytes over IPv4; set MTU = 1420 for typical 1500 paths and lower for PPPoE/tunnel stacks. Wrong MTU causes hangs, not total failure.
• Firewall the tunnel interface itself: peers can reach whatever the host allows on that interface. Restrict with nft input rules on wg0.
• SSH tunnels die silently; use ServerAliveInterval, ExitOnForwardFailure yes, and a supervisor (systemd/AutSSH) for persistent tunnels.
• Do not route production traffic through ad-hoc tunnels; use them to diagnose and migrate, not as permanent infrastructure.

Proxy Debugging

env | grep -i proxy
curl -v --proxy http://proxy:3128 https://example.com/
curl --noproxy '*' https://internal.example/
openssl s_client -proxy proxy:3128 -connect example.com:443

• http_proxy, https_proxy, no_proxy (and HTTP_PROXY variants) are honored by many tools but not all; curl, wget, git, package managers, and language runtimes each have their own precedence. Check per tool.
• NO_PROXY matching is suffix-based and inconsistent; list hosts explicitly when in doubt.
• HTTPS through a proxy uses CONNECT host:443; failures show up as 403/407 or tunnel resets in curl -v. A TLS error through a proxy often means TLS interception with an untrusted CA.
• Corporate TLS inspection requires installing the interception CA in the system trust store and every runtime's bundle (Java cacerts, Python certifi, Node NODE_EXTRA_CA_CERTS). This is a frequent cause of "works in browser, fails in code".
• Diagnose with mitmproxy/proxychains locally; verify what the proxy sees by capturing with tcpdump on the proxy path.
• Authenticated proxies need credentials; avoid embedding them in env vars on shared hosts, prefer credential helpers or per-tool config files with 600 permissions.

Anti-Patterns

• Restarting the network stack (systemctl restart NetworkManager) as a first step; it drops your remote session and hides evidence.
• iptables -F on a remote host without a rollback plan or out-of-band console.
• Turning off the firewall to prove a theory instead of capturing traffic.
• Disabling TLS verification (curl -k, NODE_TLS_REJECT_UNAUTHORIZED=0) in production code.
• Editing resolv.conf by hand on hosts where systemd-resolved or DHCP overwrites it.
• chmod 666 on /etc/resolv.conf, or installing wildcard certificates to dodge per-host names.
• Trusting ping alone: ICMP may be filtered while services work, or vice versa.
• Leaving tcpdump running unbounded or capturing to a full disk.
• Full-tunnel VPNs pushed to servers without exempting management paths.

Checklist

• [ ] Diagnose in layers; record evidence at each step before changing anything.
• [ ] DNS: correct resolver path verified (resolvectl/scutil/ipconfig), TTLs planned before migrations, split-horizon tested from both sides.
• [ ] TLS: chain and SAN validated, expiry monitored 30/14/7 days, TLS 1.2+ only.
• [ ] Firewall: default drop, established/related accepted, rules persisted and diffed.
• [ ] Remote access: keys only, tunnels supervised with keepalives.
• [ ] MTU verified for tunnels/VPN (no silent large-packet black holes).
• [ ] Proxy settings exported per tool, not assumed universal; NO_PROXY explicit.
• [ ] Captures saved as pcap for evidence, bounded in size and time.
• [ ] Changes to routing/firewall done with console access and a rollback path.

Related: Linux administration for persistence and units, performance for throughput work, hardening for firewall and remote-access defaults.

Storage and Filesystems

Planning, operating, and recovering storage: filesystem selection, NVMe, RAID, LVM, ZFS/btrfs, backups, and capacity/inode monitoring.

Filesystem Decision Table

| Need | Choice | Why | Watch out | |---|---|---|---| | General-purpose root/data on Linux | ext4 | Mature, predictable, online resize up | No built-in checksums/snapshots | | Large files, parallel I/O, big filesystems | XFS | Scales well, online grow, good allocator | Cannot shrink; needs ftype=1 for overlayfs | | Snapshots, subvolumes, compression | btrfs | Integrated CoW, easy rollback | Free-space accounting, balancing, RAID5/6 caveats | | Data integrity, compression, send/recv replication | ZFS | End-to-end checksums, ARC, datasets | RAM needs, licensing/distro packaging, no easy shrink | | Shared network storage | NFS/SMB | Multi-client convenience | Locking, permissions mapping, single point of failure | | Databases | Depends: ext4/XFS local NVMe; ZFS with tuned recordsize | Latency and fsync semantics dominate | Disable CoW per-dataset only when justified | | Object storage | S3-compatible or Ceph RGW | Durability at scale | Not a filesystem; app changes required |

• Choose per dataset, not per host: a ZFS/btrfs pool can host an ext4-formatted VM image or an XFS volume if that is what the workload wants.
• Do not enable ZFS dedup casually; it is RAM-hungry and usually slower than expected. Compression (lz4/zstd) is cheap and almost always worth it.
• btrfs RAID5/6 have historically been unsafe for certain failure modes; verify current upstream status before relying on them. Prefer RAID1/RAID10 with btrfs.
• Windows servers: NTFS remains standard; ReFS offers integrity streams and block cloning for virtualization. Verify feature support on your release.

Block Devices and NVMe

lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINTS,MODEL
blkid; findmnt --verify
nvme list; nvme smart-log /dev/nvme0
nvme error-log /dev/nvme0 | head -50
smartctl -a /dev/nvme0; smartctl -a /dev/sda
hdparm -I /dev/sda | grep -i trim
fstrim -av                             # weekly is typical; check systemd fstrim.timer

| Metric | Where | Warning sign | |---|---|---| | Percentage used | nvme smart-log | > 80-90% of rated endurance | | Media errors | smartctl -a / nvme error-log | Any growth | | Available spare | NVMe SMART | Collapsing spare pool | | Reallocated sectors | SATA SMART | Non-zero growth | | Temperature | SMART | Above vendor spec under load | | Unsafe shutdowns | NVMe | Power-loss exposure; check PSU/UPS |

• fstrim.timer (systemd) or a cron job handles periodic TRIM; online discard (discard mount option) adds per-delete latency and is generally not recommended on busy servers.
• NVMe wear is uneven with small random writes; over-provisioning and steady-state testing matter for write-heavy databases.
• Device names (/dev/nvme0n1, /dev/sda) are not stable; use UUID=/LABEL= in /etc/fstab or /dev/disk/by-id/.
• SMART self-tests do not prove a drive is healthy; monitor growth trends, not just pass/fail. Centralize SMART metrics.
• Plan for die/device failure: NVMe devices fail as whole units, and firmware bugs are real; keep firmware current per vendor advisories.

Partitioning

parted -s /dev/sdb mklabel gpt
parted -s /dev/sdb mkpart primary 1MiB 100%
sgdisk -n 1:2048:+512M -t 1:ef00 /dev/sdb     # EFI system partition
wipefs -a /dev/sdb                             # clear old signatures before reuse

• GPT is the standard; keep a 1 MiB alignment for performance and a small BIOS boot or EFI partition where needed.
• wipefs -a before reusing disks avoids stale RAID/LVM/filesystem signatures that make the kernel assemble ghosts.
• Partition tables are metadata: back up with sgdisk --backup before risky operations.
• Whole-disk use (no partition table) is normal for ZFS vdevs and many LVM PVs; match the tool's recommendation.

LVM

pvs; vgs; lvs -a -o +devices
pvcreate /dev/sdb1; vgcreate data /dev/sdb1
lvcreate -L 100G -n appdata data
lvextend -r -L +50G /dev/data/appdata        # -r resizes the filesystem too
lvcreate -s -L 10G -n appdata-snap /dev/data/appdata
lvs -S 'lv_name=~snap'

| Feature | Use | Limit | |---|---|---| | Linear LV | Simple aggregation of disks | One disk failure can lose the LV | | Striping | Throughput across disks | Worse failure exposure; no redundancy | | Mirroring | Simple redundancy | Costly; RAID tools may be better | | Snapshots (thick) | Short-lived backup windows | Full-size CoW penalty as they fill | | Thin pools | Overprovisioning, many snapshots | Pool exhaustion takes down all volumes | | Cache (SSD) | Warm data acceleration | Complexity; tiering needs monitoring | | pvmove | Migrate extents off a disk | Slow; run in maintenance windows |

• Set -r when extending so the filesystem grows in the same command; XFS growth is online, ext4 supports online grow.
• Snapshot space is consumed by writes to the origin, not by the snapshot copy; a full snapshot invalidates itself. Always monitor snapshot fill.
• Thin pools expose data_percent and metadata_percent; both can kill the pool. Alert well before 100% and never let metadata exceed ~75%.
• Do not shrink: shrinking LVs/filesystems is error-prone; restore from backup instead.
• LVM on top of mdadm is common; LVM RAID (lvconvert --type raid1) is an alternative with tighter integration but fewer recovery tools.

RAID

cat /proc/mdstat
mdadm --detail /dev/md0
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
mdadm /dev/md0 --add /dev/sdd1
echo check > /sys/block/md0/md/sync_action   # scrub

| Level | Redundancy | Capacity | Use | |---|---|---|---| | 0 | None | N | Scratch/performance only | | 1 | 1 disk | N/2 | OS disks | | 5 | 1 disk | N-1 | Budget bulk; slow rebuilds | | 6 | 2 disks | N-2 | Bulk with larger disks | | 10 | 1 per mirror pair | N/2 | Databases and busy workloads |

• Rebuilds are read-intensive and long; degraded arrays have a second-failure window. Never rebuild large arrays without a tested backup from the last 24 hours.
• Regular scrubs catch latent bad blocks before a rebuild needs them. Schedule monthly for HDD, more often for NVMe if supported.
• RAID is not backup: it protects availability, not accidental deletion or corruption.
• Hardware RAID controllers hide errors; monitor with vendor tools and keep batteries/ cache modules healthy. Prefer JBOD + software RAID/ZFS where practical.
• ZFS RAIDZ and btrfs RAID are filesystem-integrated; do not stack mdadm under them.

ZFS

zpool create tank mirror /dev/disk/by-id/nvme-A /dev/disk/by-id/nvme-B
zpool status -v; zpool list -v; zfs list -o space
zfs set compression=zstd ashift=12 atime=off tank
zfs create -o recordsize=16K tank/db
zfs snapshot tank/data@2026-09-22
zfs send -i tank/data@old tank/data@new | ssh host zfs receive backup/data
zpool scrub tank
zpool trim tank

| Property | Guidance | |---|---| | ashift=12 | 4K sectors; set at pool creation, cannot change later | | recordsize | 128K general; 8-16K for databases; 1M for large media | | compression | lz4 or zstd; nearly free, improves write amplification | | atime=off | Fewer writes; only enable where required | | xattr=sa | Better performance/metadata for ACL workloads | | sync=standard | Keep; sync=disabled risks data on power loss | | copies=2 | Metadata resilience on single-disk pools |

• ARC is RAM-hungry by design; size with zfs_arc_max and monitor hit rate, not just free RAM. Linux ARC interacts with the page cache poorly if over-limited.
• Scrub monthly; watch zpool status for CKSUM, READ, WRITE, CKS errors and replacement in progress. zpool status -v lists damaged files.
• Dedup tables are catastrophic to lose; if used, mirror the dedup device and monitor its size via zdb -DD. Default answer remains no.
• Snapshots are not backups: they share the pool and disappear with it. Use zfs send/recv or a tool like sanoid/syncoid for real replication.
• Pool capacity above ~80% degrades allocation; keep headroom and alert earlier.
• Verify distro packaging/version; feature flags limit pool portability across releases.

btrfs

mkfs.btrfs -L data /dev/sdb
btrfs subvolume create /mnt/data/@app
btrfs subvolume snapshot -r /mnt/data/@app /mnt/data/snapshots/app-$(date +%F)
btrfs filesystem usage -T /mnt/data
btrfs scrub start -B /mnt/data
btrfs device stats /mnt/data
btrfs subvolume snapshot -r / /mnt/snapshots/root-$(date +%F)

• Layout with subvolumes (@, @home, @snapshots) enables rollback of root without touching user data. Mount subvolumes explicitly in /etc/fstab with subvol=.
• btrfs filesystem df/usage shows data, metadata, and system chunks separately; "df says 50% but writes fail" is usually metadata exhaustion.
• Balancing reclaims unevenly allocated chunks; btrfs balance start -dusage=50 is a targeted approach. Unfiltered full balances are long and can be risky on full disks.
• Scrub verifies checksums and repairs from mirrors; monitor device stats for growing read/write/checksum errors.
• Send/receive replicates snapshots; incremental streams are efficient but the parent-child chain must be intact.
• RAID1 for metadata and data is the safe choice; avoid RAID5/6 without verifying current upstream guidance.

Mount Options and Tuning

| Mount option | Effect | When | |---|---|---| | noatime | No access-time writes | Almost always on servers | | relatime | Coarse atime (kernel default) | Default is fine | | discard | Online TRIM | Avoid; use fstrim.timer | | nofail | Boot continues if missing | Data/external volumes | | x-systemd.automount | Mount on first access | Removable/occasional paths | | compress=zstd (btrfs) | Transparent compression | CPU headroom available | | errors=remount-ro (ext4) | Fail safe after I/O errors | Default hardening |

• Test /etc/fstab changes with findmnt --verify and mount -a before rebooting; a broken fstab lands the host in emergency mode.
• nofail plus x-systemd.device-timeout= prevents boot hangs on missing SAN/NFS.
• NFS tuning: vers=4.2, hard, timeo, retrans, noatime; soft mounts risk silent corruption. Use autofs for on-demand mounts.
• Network filesystems need their own timeout and reconnect strategy; an unreachable NFS server can hang processes uninterruptibly (D state) and block shutdown.

Backups: 3-2-1 and Restore

| Layer | Example | Frequency | |---|---|---| | Local snapshot | ZFS/btrfs/LVM snapshot | Hourly for fast rollback | | Local backup repository | restic/borg to attached disk | Daily | | Offsite copy | Object storage via restic/rclone | Daily, encrypted | | Immutable/versioned | Object lock, append-only repo | Retention per policy | | Restore test | Full or sampled restore | Quarterly |

restic -r s3:s3.example.com/backups init
restic -r s3:s3.example.com/backups backup /srv --exclude-caches
restic snapshots; restic check --read-data-subset=5%
restic restore latest --target /tmp/restore --include /srv/app
borg init --encryption=repokey-blake2 ssh://backup@host/./repo
borg create ::'{host}-{now}' /srv; borg check --verify-data

• The 3-2-1 rule: 3 copies, 2 different media, 1 offsite. Modern practice adds "1 offline/immutable" for ransomware resilience.
• Encryption keys are the single point of failure for encrypted repos: escrow them separately from the backups. Losing the passphrase loses the data.
• Versioning/retention: prune with grandfather-father-son policies and test that pruned data is no longer needed. Untested retention is a guess.
• Snapshots on the same pool/array are not backups; corruption and deletion propagate.
• Monitor backup job exit status and staleness, not just "the job ran". Alert if the last successful snapshot is older than the policy.
• Restore drills: restore a real service and verify data checksums. Document RTO/RPO and test against them.
• Database backups need consistency: dump with pg_dump/mysqldump/.backup or filesystem snapshots coordinated with the DB, not raw file copies of live files.
• Immutability: object lock, append-only Borg repos, or offline rotation defeat credential theft. Test that deletion actually fails.

Capacity and Inode Monitoring

df -h; df -i
findmnt -o TARGET,FSTYPE,SIZE,USED,AVAIL,USE%
du -xhd1 /var | sort -h | tail
ncdu /var                              # interactive offline analysis
lsof +L1 | head                        # deleted-but-open files holding space

• Alert on both space and inode percentage; inode exhaustion breaks writes while df -h shows free space (classic on mail spools, session dirs, and many-small-files workloads).
• Reserved blocks (tune2fs -m 1) matter on large ext4 volumes; 5% of 10 TB is 500 GB.
• Deleted-but-open files keep space allocated; find with lsof +L1 and restart the holder rather than rebooting.
• Thin pools and snapshots fail writes before the underlying device is full; monitor the abstraction (vgs --units g, zpool list, btrfs filesystem usage), not raw df.
• Growth forecasting: track daily delta and alert on time-to-full, not just threshold. Sudden log growth usually means a runaway service or verbose debug flag.
• Quotas (project quotas on XFS/ext4 with prjquota, ZFS refquota, btrfs qgroup) contain noisy tenants; test enforcement before production.

Anti-Patterns

• RAID as a backup strategy; snapshots as offsite backups.
• dd from a live database for backups without consistency controls.
• Extending a filesystem without checking the underlying LV/pool/array free space.
• Full btrfs balance or md rebuild on a Sunday morning "to be safe".
• ZFS dedup without RAM sizing and mirrored dedup devices.
• Ignoring SMART growth and NVMe percentage-used until failure.
• Running databases on NFS without verifying fsync semantics and vendor support.
• Storing encryption keys next to the backup repository.
• Letting root or /var reach 100%; reserving blocks and alerting at 80%.
• Using discard mount option on busy databases instead of scheduled fstrim.

Checklist

• [ ] Filesystem choice documented per workload with rationale.
• [ ] Stable device identifiers (by-id/UUID) used in fstab and pool configs.
• [ ] fstrim.timer (or equivalent) active; NVMe SMART monitored centrally.
• [ ] RAID/LVM/ZFS/btrfs configured with redundancy appropriate to the data.
• [ ] Scrubs scheduled; pool/array state alerted.
• [ ] Snapshots defined with retention; not confused with backups.
• [ ] 3-2-1 backups verified by restore drills; keys escrowed separately.
• [ ] Space AND inodes monitored with growth forecasting on every filesystem.
• [ ] fstab changes validated with findmnt --verify; nofail on non-boot volumes.
• [ ] Database backup method matches consistency requirements.

Related: Linux administration for units and timers, performance for I/O saturation, and containers for image/layer storage.

Performance Tuning

Measuring and improving host performance: perf, eBPF/bpftrace, classic counters, PSI, ulimits, sysctl tuning, memory pressure, and the OOM killer.

Method Before Tools

1. Define the symptom — latency, throughput, errors, or saturation, with a target (p99 < 50 ms, 10k rps) and a time window.
2. Check utilization and saturation — CPU, memory, disk, network, and file descriptors. The USE method (utilization, saturation, errors per resource) finds the bottleneck quickly.
3. Measure at the right layer — cgroup/container, process, host, or fleet. A noisy neighbor is invisible from inside the container.
4. Establish a baseline — record before-and-after numbers under comparable load.
5. Change one variable, re-measure, and roll back if the change does not help.

• Tuning without measurement is superstition; most sysctl folklore is workload- and kernel-version-specific.
• The bottleneck moves: fix CPU and the disk becomes the limit. Expect to iterate.
• Prefer architectural fixes (caching, batching, async, capacity) over parameter tuning.
• Use observability practices for long-term visibility; this reference is for interactive diagnosis and tuning.

Load and Saturation Basics

uptime; cat /proc/loadavg
nproc; lscpu | head -20
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
mpstat -P ALL 1 5
pidstat -u -r -d 1 5

• Load average counts runnable + uninterruptible (D-state) tasks. With 8 CPUs, a load of 8 is saturated; a load of 4 with high I/O wait may still be unhealthy because D-state tasks inflate it.
• PSI (pressure stall information) is the best single saturation signal: some = at least one task stalled, full = all tasks stalled. Sustained full on memory or IO means real user-visible pain.
• vmstat 1: watch r (run queue), b (blocked), si/so (swap), wa (I/O wait), and st (stolen by hypervisor). Stolen time means the host is oversubscribed.
• CPU frequency and thermal throttling can cap performance invisibly: cpupower frequency-info, turbostat, sensors.

perf and ftrace

perf stat -p PID sleep 5
perf top -g
perf record -F 99 -g -p PID -- sleep 30
perf report --stdio --sort=comm,dso,symbol | head -40
perf trace -p PID
perf sched latency
perf stat -e cache-misses,cache-references,cycles,instructions -p PID sleep 5

• perf record -F 99 -g samples at 99 Hz with call graphs; 30 seconds of samples is enough for a flame graph (perf script | stackcollapse-perf.pl | flamegraph.pl).
• Userspace symbols need -g at build time and debuginfo packages for accurate frames; stripped binaries show only addresses.
• Off-CPU and wakeup analysis: perf sched, offcputime (BCC), and perf trace show where threads block, complementing on-CPU profiles.
• perf stat counter ratios diagnose microarchitecture issues: high cache-misses per reference, low IPC, or branch-miss spikes point at data layout or branchy code.
• Tracepoints (perf list) are stable; kprobes are kernel-version-sensitive. Prefer tracepoints and USDT probes in automation.
• Lock contention: perf lock, futex tracepoints, and bcc tool offwaketime.

eBPF / bpftrace

# top functions by CPU, 5s
bpftrace -e 'profile:hz:99 /pid == $1/ { @[ustack] = count(); }' 1234

# I/O latency histogram by process
bpftrace -e 'tracepoint:block:block_rq_complete { @us[comm] = hist(args->nr_sector == 0 ? 0 : 0); }'

# syscall counts per process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[pid, comm] = count(); } interval:s:10 { print(@); clear(@); }'

# open failures
bpftrace -e 'tracepoint:syscalls:sys_exit_openat /args->ret < 0/ { @[comm, args->ret] = count(); }'

| Tool set | Strength | Use | |---|---|---| | bpftrace | Ad-hoc one-liners, histograms | Interactive diagnosis | | BCC tools (execsnoop, opensnoop, biolatency, tcplife) | Ready-made scripts | Common questions fast | | libbpf + CO-RE | Portable production agents | Vendor/monitoring integration | | perf + ftrace | Built-in, no extra packages | When eBPF is unavailable |

• bpftrace needs root or CAP_BPF/CAP_PERFMON; check kernel.unprivileged_bpf_disabled and distro policy.
• Kernel version and BTF availability determine which probes work: BTF-enabled kernels (mainline distro kernels) support CO-RE; older/custom kernels may not.
• Keep probes bounded: aggregate into maps, avoid printing per-event in hot paths, and set interval/timeout to end collection.
• Overhead is usually low but not zero; measure with and without probes on tight-SLO systems.
• eBPF is not a substitute for application tracing; use OpenTelemetry for request-level causality.

Classic Counters

iostat -xz 1 5
vmstat -w 1 5
pidstat -u -r -d -w 1 5
sar -n DEV 1 5
iotop -oPa
free -h; cat /proc/meminfo | head -20

| Tool | Field | Interpretation | |---|---|---| | iostat | r_await/w_await | Per-request latency (ms); the real latency signal | | iostat | aqu-sz | Queue depth; growing means saturation | | iostat | %util | Misleading on NVMe/RAID; near 100% only meaningful for single HDD | | vmstat | r, b | Run queue and blocked tasks | | vmstat | si/so | Swap in/out; sustained non-zero is memory pressure | | pidstat | %CPU, %MEM | Per-process attribution | | iotop | IO> | Live per-process I/O rates |

• Compare latency percentiles, not averages. %iowait is a CPU idle state accounting artifact, not a disk metric; use it only as a hint.
• Queue depth thresholds depend on the device: SATA/NVMe have very different concurrency capabilities. Latency is the universal measure.
• Use blktrace/btt for deep block-layer analysis, biolatency for histograms.
• File-descriptor and connection counts are resources too: lsof -p PID | wc -l, ss -s; exhaustion looks like an application hang.

Memory: Pressure and Reclaim

free -h
grep -E 'MemAvailable|Dirty|Writeback|SwapCached' /proc/meminfo
cat /proc/pressure/memory
cat /sys/fs/cgroup/<path>/memory.current /sys/fs/cgroup/<path>/memory.stat
perf stat -e page-faults -p PID sleep 5
slabtop -s c

• free vs available: Linux uses free RAM for page cache; MemAvailable is the number that matters. "Used" memory near 100% is normal and healthy.
• Reclaim pressure shows as growing PSI some/full on memory, rising pgscan/pgsteal, and eventually swap activity. Treat those as the alarm.
• Page cache can be dropped only for measurement, not as a fix: echo 3 > /proc/sys/vm/drop_caches (privileged, global, harmful to performance).
• Dirty page writeback: large Dirty values cause latency spikes when the kernel flushes. Tune vm.dirty_ratio/vm.dirty_background_ratio only with evidence; modern kernels auto-tune.
• Slab growth (especially dentry/inode) indicates filesystem churn; slabtop and drop_caches diagnostics, not a steady-state fix.
• Leaks: RSS trending up with flat workload; use smem/pmap, heap profilers, and /proc/PID/smaps_rollup. OOM kills often follow a slow leak, not a sudden spike.

OOM Killer

dmesg -T | grep -i -E 'oom|killed process'
cat /sys/fs/cgroup/<path>/memory.events
cat /proc/<pid>/oom_score; cat /proc/<pid>/oom_score_adj
systemctl show -p MemoryMax,MemoryHigh,OOMScoreAdjust myapp.service

• Global OOM: the kernel picks a victim by oom_score (RSS + adjustments). Inside a cgroup, the cgroup limit is enforced first and the kill is recorded in memory.events.
• oom_score_adj ranges -1000 (never kill) to 1000 (kill first). Use sparingly and document; protecting everything protects nothing.
• systemd services: OOMScoreAdjust=, MemoryMin=, MemoryLow=, MemoryHigh=, MemoryMax=, ManagedOOMMemoryPressure= provide policy beyond the raw killer.
• Diagnose cause before tuning: a container killed at its limit needs more memory or fewer allocations, not MemoryMax=infinity.
• Kernel OOM messages include the cgroup and oom-kill constraints; capture them in monitoring before journal rotation loses them.
• Swap is not a fix for leaks; it masks growth and converts memory pressure into I/O stalls. Configure swap for hibernation/headroom, not as capacity.

Limits: ulimits and systemd

ulimit -a
cat /proc/<pid>/limits
prlimit --pid <pid>
systemctl show myapp.service -p LimitNOFILE -p LimitMEMLOCK -p TasksMax
cat /etc/security/limits.conf; cat /etc/security/limits.d/*.conf

| Limit | Symptom when hit | Notes | |---|---|---| | nofile | too many open files | Raise per service, not globally by default | | nproc | fork/thread failures | cgroup pids.max also applies | | memlock | eBPF/DPDK/RDMA failures | Needs explicit raise; systemd LimitMEMLOCK=infinity | | core | No crash dump | Set core_pattern deliberately | | stack | Thread creation failures | Rarely the real problem | | max user processes | Login failures under load | Distinguish from cgroup caps |

• systemd services: limits must be set in the unit (LimitNOFILE=, TasksMax=); limits.conf is ignored for services, which do not go through PAM.
• TasksMax maps to cgroup pids.max and is the correct fork-bomb control.
• Never set nofile to unlimited globally on busy hosts; it can exhaust kernel memory. Size per service from actual lsof | wc -l headroom.
• Inherited limits: a process forked from a shell with raised limits keeps them; this hides misconfiguration until a service starts from systemd.

sysctl Tuning

sysctl -a | grep -E 'swappiness|somaxconn|tcp_congestion'
sysctl net.core.somaxconn vm.swappiness vm.dirty_background_ratio
sysctl -w net.core.somaxconn=4096            # temporary
# persist in /etc/sysctl.d/99-tuning.conf after review

| Key | Typical direction | Cautions | |---|---|---| | net.core.somaxconn | Raise for busy accept queues | Apps must also use a large backlog | | net.ipv4.tcp_max_syn_backlog | Raise for SYN floods | Interacts with syncookies | | net.ipv4.ip_local_port_range | Widen for high outbound rates | TIME-WAIT exhaustion is the real cause | | net.ipv4.tcp_tw_reuse | Often safe for outbound clients | Do not use tcp_tw_recycle (removed) | | vm.swappiness | Lower (1-10) to prefer reclaim | Swapping is still needed under pressure | | vm.dirty_ratio/dirty_background_ratio | Lower for latency-sensitive disks | Higher values improve throughput | | vm.overcommit_memory | Leave at 0 | 1 enables OOM surprises | | kernel.pid_max | Raise for many threads | Also check threads-max | | net.netfilter.nf_conntrack_max | Raise for NAT/firewall loads | Watch nf_conntrack_count | | vm.max_map_count | Raise for some databases/JVMs | Elasticsearch/Neo4j historically needed this |

• Every change is a hypothesis: document the key, reason, value, date, and measurement.
• sysctl.d fragments are the persistent mechanism; avoid editing /etc/sysctl.conf directly on managed hosts.
• Container hosts: many keys are namespaced per network namespace; setting them inside a container may not affect the host or may be rejected.
• THP (transparent_hugepage) defaults changed across kernels; do not blanket-disable or force always. Measure with the actual workload (/sys/kernel/mm/transparent_hugepage/).
• NUMA: pin latency-critical processes with numactl --cpunodebind --membind; check numastat -p PID for remote memory access before blaming the app.
• CPU governors: performance trades power for consistent latency; schedutil is usually the right default on modern kernels.

Profiling Workflow (recipes)

| Symptom | First check | Then | |---|---|---| | High CPU, app-level | perf top -g, flame graph | Optimize hot symbols, check IPC | | High CPU, kernel-level | perf top -g -K, mpstat | Syscall/os noise, softirqs, network stack | | Latency with idle CPU | offcputime, perf sched | Lock contention, blocking I/O | | Slow disk writes | iostat -xz, biolatency | Queue depth, fsync pattern, write amplification | | Slow network | ss -i, mtr, perf stat -e net:* | RTT/loss, retransmits, buffer sizes | | Periodic stalls | vmstat/PSI over time, ftrace | Writeback, timer storms, cron contention | | Container throttled | cpu.stat, memory.events | Raise limits or reduce work |

• Record commands and outputs in the incident/ticket; tuning without evidence is unverifiable later.
• Benchmark with representative load; microbenchmarks mislead for I/O and network.
• Roll back every tuning change that does not show measurable improvement.

Anti-Patterns

• %util at 100% treated as "disk is dying" on NVMe arrays.
• Raising nofile globally to hide an application FD leak.
• vm.swappiness=0 on a machine that genuinely needs to reclaim anonymous memory.
• Copy-pasting sysctl sets from blog posts without measurement or documentation.
• echo 3 > drop_caches as a production remedy.
• Tuning the container while the host/neighbor is saturated.
• Using load average alone to declare CPU saturation on many-core hosts.
• Ignoring PSI and waiting for OOM kills before acting on memory pressure.
• Profiling release binaries without symbols and trusting the resulting graph.

Checklist

• [ ] Symptom quantified (metric, target, window) before any change.
• [ ] Baseline captured; one variable changed per iteration.
• [ ] PSI checked for CPU, memory, and I/O saturation.
• [ ] Bottleneck attributed to host, cgroup, or process correctly.
• [ ] Limits reviewed in units (LimitNOFILE, TasksMax), not just ulimit.
• [ ] OOM events correlated with cgroup limits and growth trends.
• [ ] Every sysctl/unit change documented with reason and measurement.
• [ ] Changes reverted unless they measurably help.
• [ ] Monitoring/alerts updated for the saturation signal you found.

Related: Linux administration for cgroups and units, storage for I/O behavior, and networking for transport diagnosis.

Containers from the OS Layer

Understanding and debugging containers as operating-system constructs: namespaces, cgroups, OCI runtimes, rootless mode, overlay filesystems, and host-side debugging.

What a Container Is

A container is a process (or process tree) with:

1. Namespaces restricting what it can see (PID, mount, network, UTS, IPC, user, cgroup, time).
2. cgroups limiting what it can consume (CPU, memory, pids, I/O).
3. A root filesystem assembled from image layers.
4. Security policy — capabilities, seccomp, LSMs (SELinux/AppArmor), and user mapping.

There is no container kernel. runc/crun are thin process launchers; containerd, Docker, and Podman manage images, networks, and lifecycle on top. This is why host-side debugging tools work directly on containers.

unshare --user --map-root-user --pid --fork --mount-proc bash   # the essence, no runtime
lsns -p <pid>

Namespaces in Practice

| Namespace | Flag | What the container sees | |---|---|---| | pid | CLONE_NEWPID | Itself as PID 1; host PIDs hidden | | mount | CLONE_NEWNS | Its own root and mounts | | net | CLONE_NEWNET | Own interfaces, routes, ports, loopback | | uts | CLONE_NEWUTS | Own hostname/domain | | ipc | CLONE_NEWIPC | Own SysV/POSIX IPC objects | | user | CLONE_NEWUSER | UID/GID mapping; rootless foundation | | cgroup | CLONE_NEWCGROUP | Sees cgroup paths relative to itself | | time | CLONE_NEWTIME | Clock offsets per process group |

# which namespaces does a container process use?
pid=$(docker inspect -f '{{.State.Pid}}' web)
ls -l /proc/$pid/ns/
nsenter -t "$pid" -a -- hostname
nsenter -t "$pid" -n -- ss -tulpn
nsenter -t "$pid" -m -- ls /

• Container network debugging: nsenter -n gives the container's routing table, iptables, and sockets from the host, without installing tools inside the image.
• File-descriptor tricks work across namespaces: docker inspect/crictl inspect → host PID → /proc/<pid>/root/ and /proc/<pid>/cwd/.
• PID 1 semantics inside containers matter: it must reap zombies and forward signals. Use --init or a proper init if the app does not.
• Network namespaces are per-container; host ss -tulpn does not show container listeners (they live in their own netns or are DNAT'd by the runtime).

cgroups v2 and Limits

docker run --memory=512m --memory-swap=512m --cpus=1.5 --pids-limit=256 nginx
podman run --memory=512m --cpus=1.5 --read-only --cap-drop=ALL nginx
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.stat

| Flag | cgroup file | Effect | |---|---|---| | --memory | memory.max | Hard cap; OOM-kill inside the cgroup | | --memory-reservation | memory.low | Soft protection under pressure | | --cpus | cpu.max | Bandwidth cap (quota period) | | --cpu-shares | cpu.weight | Relative share, no hard cap | | --pids-limit | pids.max | Fork-bomb containment | | --blkio-weight | io.weight | Relative I/O share | | --device-read-bps | io.max | Hard I/O ceiling |

• cgroups v2 is unified on current distros; cpu.stat exposes nr_throttled and throttled_usec — a throttled container looks slow while host CPU is idle.
• Memory: memory.events (oom, oom_kill, high) distinguishes limit kills from global OOM; memory.current/memory.peak show usage trends.
• JVM/Node/Go runtimes often need heap limits tuned to the cgroup limit; container-aware flags exist (-XX:MaxRAMPercentage, Node --max-old-space-size) — verify per runtime.
• --memory-swap equal to --memory disables swap for the container; otherwise the difference is the allowed swap.
• Kernel OOM inside a namespace still kills the largest task in the cgroup; identify it with memory.oom.group settings and logs.

OCI Runtime and Runtimes

containerd --version; runc --version; crictl info
ctr -n k8s.io containers list
ctr -n k8s.io tasks list
crictl ps -a; crictl logs <id>; crictl inspect <id> | jq '.info.runtimeSpec'
runc list

| Component | Role | |---|---| | OCI image spec | Tarball layers plus a JSON config (entrypoint, env, ports) | | OCI runtime spec | config.json: namespaces, mounts, capabilities, seccomp | | runc | Reference runtime; creates namespaces and execs the process | | crun | C alternative; lighter and faster startup | | containerd | Image/layer store, snapshotters, shim lifecycle (Kubernetes CRI) | | Docker Engine | Developer UX over containerd (or its own daemon paths) | | Podman | Daemonless CLI; rootless-first; pods compatible with Kubernetes YAML |

• runc spec generates a config.json; runtimes are swappable and the RuntimeClass in Kubernetes can select runsc (gVisor) or Kata for stronger isolation.
• crictl talks to the CRI socket (Kubernetes nodes); ctr talks to containerd directly (-n namespace matters); nerdctl is a friendlier containerd CLI.
• Docker vs containerd: docker ps may not show CRI-managed containers and vice versa; use crictl on Kubernetes nodes and docker on developer hosts.
• Runtime hooks (prestart, createRuntime) modify container setup; they run as root and are a privilege-escalation surface. Audit hooks in multi-tenant clusters.
• The shim (containerd-shim) survives daemon restarts and holds the container's stdio and exit status; stuck shims are a real failure mode — inspect ctr/crictl, not just the orchestrator.

Rootless Containers

cat /etc/subuid /etc/subgid                 # user:start:count ranges
podman info | grep -A5 rootless
podman run --rm -p 8080:80 nginx            # >=1024 ports work by default
sysctl net.ipv4.ip_unprivileged_port_start  # lower to allow <1024 in rootless netns
podman system migrate                        # refresh mappings after subuid changes

| Requirement | Detail | |---|---| | subuid/subgid | /etc/subuid, /etc/subgid (or useradd defaults) must grant ranges | | newuidmap/newgidmap | Setuid helpers from uidmap package map ranges | | User namespaces | kernel.unprivileged_userns_clone=1 where distros gate it | | Storage driver | overlayfs native where possible; fuse-overlayfs fallback | | Networking | rootlesskit + slirp4netns (userspace) or pasta; slower than veth | | cgroup delegation | systemd lingers + Delegate=yes for limits in user slices | | Ports | <1024 needs ip_unprivileged_port_start or CAP_NET_BIND_SERVICE |

• Rootless pods/containers run mapped to subordinate UIDs; a container root escape lands as an unprivileged host UID. This is the main security benefit, not a full sandbox.
• podman generate systemd is deprecated in favor of Quadlet (.container files under /etc/containers/systemd/); verify the current supported approach upstream.
• Rootless storage can exhaust ~/.local/share/containers; monitor it like any disk.
• podman unshare starts a shell in the user namespace to inspect/manage mapped files.
• Rootless limitations: no host networking by default, no arbitrary netfilter rules, different performance profile. Test before standardizing.

Overlay Filesystems and Layers

docker info | grep -i storage
cat /proc/mounts | grep overlay
mount | grep overlay
du -sh /var/lib/docker/overlay2 /var/lib/containers/storage
docker system df; docker image prune -a; docker builder prune

• overlayfs unions a read-only lowerdir stack with a writable upperdir; the first write to a file triggers copy-up (whole file copied), which can be expensive for large files.
• The backing filesystem must support d_type (XFS with ftype=1, ext4). Overlay on top of network filesystems (NFS) or btrfs subvolumes is unsupported or problematic.
• Layer size is not additive: shared base layers are stored once. Tooling that sums image sizes overstates disk use.
• Deleted files leave whiteouts; frequent rebuilds accumulate layers. Prune images, build caches, and volumes on a schedule, and monitor the state directory.
• Inode exhaustion is a common container-host failure when many small files and layers exist; df -i on the state filesystem is mandatory.
• Use volumes or bind mounts for data; writes to the container layer are ephemeral, slower, and bloat the image store.
• Rootless storage may use fuse-overlayfs, which is slower and has different semantics; verify with podman info.

Networking (runtime view)

docker network inspect bridge
iptables -t nat -L DOCKER -n --line-numbers   # legacy docker NAT chain
podman network inspect podman
crictl inspectp <pod-id> | jq '.info.runtimeSpec.linux.namespaces'

• Default bridge networks use NAT and a local DNS resolver; container names resolve only on user-defined networks (Docker), not the default bridge.
• Port publishing (-p 8080:80) is DNAT in the host netns; conflicts show as address already in use on the host ss, not inside the container.
• CNI/CNI-like plugins (Calico, Cilium) program host routing and eBPF datapaths; container connectivity problems can be host routing problems.
• network_mode: host bypasses isolation; convenient and dangerous — no port mapping, shares the host's DNS and firewall state.
• MTU mismatches between overlay networks and the underlay cause large-packet hangs; check ip link inside the container and the CNI config.
• DNS inside containers: /etc/resolv.conf is generated per network; with systemd- resolved on the host, watch for the 127.0.0.53 stub being unreachable from the container's netns.

Debugging Containers from the Host

| Question | Command | |---|---| | Host PID of container process? | docker inspect -f '{{.State.Pid}}' c / crictl inspect c | | What is it doing? | nsenter -t PID -a -- ps auxf, top -p PID | | Network state? | nsenter -t PID -n -- ss -tulpn, ip addr, ip route | | Filesystem view? | ls /proc/PID/root, nsenter -t PID -m -- ls / | | Syscalls? | strace -f -p PID -tt -T (briefly; may be blocked by seccomp) | | Kernel stack / D state? | cat /proc/PID/stack, cat /proc/PID/wchan | | CPU/memory inside cgroup? | cat /sys/fs/cgroup/.../cpu.stat, memory.current | | Distroless image (no shell)? | docker debug, kubectl debug --target, ephemeral containers | | Exit code / OOM? | docker inspect .State, crictl inspect, cgroup memory.events |

# ephemeral debug container sharing the target's namespaces (Kubernetes)
kubectl debug -it pod/web --image=nicolaka/netshoot --target=web
# Docker Desktop / CLI debug tool with a toolbox image
docker debug web

• nsenter -a covers all namespaces but uses the host filesystem view for binaries; -m alone enters the mount namespace for container filesystem inspection.
• Prefer ephemeral/debug containers over --privileged; --privileged disables the very isolation you are debugging and hides capability-related failures.
• Seccomp/AppArmor denials surface as EPERM/Operation not permitted; check dmesg/audit logs on the host, and inspect the profile in docker inspect/crictl inspect.
• A container in D state is usually blocked on I/O (disk, NFS, overlay merge); inspect /proc/PID/stack and the host's iostat.
• Reproduce with the same image digest and runtime; "works locally" often means different architecture (arm64 vs amd64) or image.

Images: Minimal and Reproducible

• Pin base images by digest for reproducibility; tag-only references are mutable.
• Distroless or scratch images shrink attack surface but remove shells — plan debugging via ephemeral containers.
• Multi-stage builds keep build toolchains out of runtime images; verify no secrets or source leaks in layers (docker history, crane export, or a scanner).
• Set USER non-root, drop capabilities, use --read-only with tmpfs where possible, and declare HEALTHCHECK/probes.
• HEALTHCHECK/liveness semantics differ between Docker and Kubernetes; do not copy one to the other without thought.
• Image scanning (Trivy, Grype, major registries) is necessary but not sufficient; rebuild to pick up base-image fixes.

Anti-Patterns

• Running --privileged to make a container start.
• Debugging by rebuilding the image with apt install/apk add instead of using ephemeral tooling.
• Mounting the Docker socket into containers; it is root-equivalent on the host.
• Storing databases in the container writable layer.
• Assuming docker ps shows everything on a Kubernetes node.
• Sizing memory limits without runtime awareness (JVM/Node heaps).
• Using latest tags in production or relying on mutable base images.
• Treating a container as a security boundary for untrusted code without a sandboxed runtime (gVisor/Kata).
• Ignoring inode usage on the container state filesystem.
• Leaving orphaned volumes, images, and networks accumulating for years.

Checklist

• [ ] Runtime and version inventory known (Docker/containerd/Podman, runc/crun).
• [ ] Resource limits set for CPU, memory, and pids on every workload.
• [ ] Runtimes container-aware or heaps explicitly sized to limits.
• [ ] Rootless evaluated where threat model or policy requires it; subuid/subgid set.
• [ ] State filesystem monitored for space AND inodes; prune schedule defined.
• [ ] Host-side debugging path documented (nsenter, ephemeral containers).
• [ ] Security posture: non-root user, dropped capabilities, seccomp/AppArmor, read-only rootfs where possible.
• [ ] Images pinned by digest, minimal, scanned, and rebuilt regularly.
• [ ] Networking mode and DNS behavior understood for each workload.
• [ ] Kubernetes nodes managed via crictl, developer hosts via Docker/Podman; not conflated.

Related: Linux administration for cgroup and systemd context, security hardening for LSM profiles and least privilege.

Security Hardening

Hardening hosts and workloads: SSH, SELinux/AppArmor, auditd, firewall defaults, CIS-style baselines, patch strategy, and least privilege.

Threat-Model First

Hardening without a threat model produces either theater or breakage. Start from:

| Question | Consequence | |---|---| | Internet-exposed or internal only? | Determines SSH, firewall, and update posture | | What data can this host reach? | Determines credential and network segmentation needs | | Who administers it and how? | Determines identity, sudo, and audit design | | What is the blast radius of compromise? | Determines isolation (VM/container/sandbox) | | What compliance regime applies? | Determines evidence and baseline requirements |

• Assume perimeter breach; segment networks so one host does not grant access to all.
• Defense in depth: patching, least privilege, mandatory access control, logging, and backups each cover the others' failures.
• Every control has an operational cost; document accepted risks and deviations rather than silently weakening controls.

Patch Strategy

| Layer | Mechanism | Cadence | |---|---|---| | OS security updates | unattended-upgrades, dnf-automatic, WUfB | Automatic for security, windowed for features | | Kernel | Distro kernel, livepatch where supported | Reboot window; livepatch covers CVEs between reboots | | Applications | Version-pinned package manager, container base rebuild | Rebuild images on base updates | | Firmware | Vendor tooling (fwupd, vendor utilities) | Per advisory | | EOL tracking | Inventory with OS/runtime versions | Alert months before EOL | | Emergency | Out-of-band critical advisories | Defined runbook and communication path |

• Patch the exploit path first: internet-facing services, then lateral-movement enablers (SSH, sudo, agents), then everything else.
• Automatic updates can break services; stage in a canary ring and monitor. Never let patching be "someone else's job".
• Reboots are part of patching: kernel, glibc, and systemd updates need them. Track pending reboot state (e.g., /var/run/reboot-required, needs-restarting -r, Get-WindowsUpdate reboot flags).
• Verify EOL dates upstream; do not extrapolate. Unsupported kernels and runtimes are the root cause of many "0-day" compromises.
• Livepatching covers only kernel CVEs and only with a supported vendor subscription; it is a bridge, not a replacement for maintenance windows.

SSH Hardening

# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
MaxAuthTries 3
MaxSessions 5
LoginGraceTime 20
ClientAliveInterval 300
ClientAliveCountMax 2
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no
PermitEmptyPasswords no
UsePAM yes
AllowGroups ssh-users
LogLevel VERBOSE

sshd -t                                   # validate before reload
systemctl reload sshd || systemctl reload ssh
ssh-keygen -t ed25519 -a 100 -C "user@host"
ssh-keygen -t ed25519-sk -C "user@host"   # hardware-backed (FIDO2)
ssh -o StrictHostKeyChecking=accept-new host
ssh-keygen -R host

| Control | Recommendation | Notes | |---|---|---| | Keys | Ed25519 (or ed25519-sk) | RSA 3072+ only for legacy; DSA is dead | | Passwords | Disabled | Exceptions documented, MFA required | | Root login | no; use sudo | prohibit-password only as a transition | | MFA | TOTP or hardware keys | AuthenticationMethods publickey,keyboard-interactive | | Agent forwarding | Off by default | Use ProxyJump instead | | Port change | Optional, low value | Reduces noise only, not a control | | fail2ban/sshguard | Optional with keys | Useful where passwords remain | | Certificates | Short-lived CA-signed keys | Scales better than authorized_keys | | Access scope | AllowGroups/Match blocks | Restrict by user, group, source, command | | Sessions | MaxSessions, timeouts | Limits hijacked connection persistence |

• ProxyJump bastion keeps the bastion as the only exposed SSH endpoint; the bastion runs no other services and logs sessions centrally.
• Host keys must be verified; distribute known_hosts via config management or use SSHFP/DNS or certificates.
• authorized_keys options matter: from=, command=, no-agent-forwarding, no-port-forwarding, no-pty, restrict (deny by default).
• Client config belongs in the repo (~/.ssh/config templates), with separate keys per role/host; never reuse one key everywhere.
• Private keys: mode 600, passphrase or hardware-backed, never copied to servers.
• Validate with sshd -t and keep a second session open while changing SSH settings; lockouts are self-inflicted outages.

SELinux (RHEL family)

getenforce; sestatus
ls -Z /var/www/html; ps -eZ | grep httpd
semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?"
restorecon -Rv /srv/www
setsebool -P httpd_can_network_connect on
ausearch -m AVC,USER_AVC -ts recent
sealert -a /var/log/audit/audit.log

| Mode | Meaning | Use | |---|---|---| | enforcing | Policy violations denied | Production default | | permissive | Logged but allowed | Diagnosis only; alerts still fire | | disabled | No policy | Only with an accepted written risk |

• Never setenforce 0 as a permanent fix; fix the label or boolean instead.
• Contexts follow files; moving files between roles (mv preserves context) breaks labels — use restorecon and define semanage fcontext rules for custom paths.
• audit2allow generates policy from denials: review every rule; blindly allowing all denials creates permissive policy and hides attacks.
• Booleans toggle supported behaviors (getsebool -a); use -P to persist.
• Relabeling (touch /.autorelabel, fixfiles -F onboot) takes a reboot and can be slow; know why labels broke (usually a copied filesystem or an unpacked archive).
• Container hosts: containers get container_t labels; volume mounts need :z/:Z or explicit context configuration.

AppArmor (Ubuntu/SUSE)

aa-status
aa-enforce /etc/apparmor.d/usr.sbin.nginx
aa-complain /etc/apparmor.d/usr.sbin.nginx
journalctl -k | grep -i apparmor
apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx

• Profiles restrict file access, capabilities, and network operations per binary; shipped profiles cover many daemons out of the box.
• complain mode logs but allows; use it to develop a profile, then enforce.
• Denials appear in the kernel journal; iterate on the profile rather than disabling AppArmor.
• Profile reload is per-profile; no reboot needed. Custom profiles live in /etc/apparmor.d/.
• For containers, the docker-default profile applies unless overridden; do not set apparmor=unconfined to make an image work.

auditd

auditctl -l
auditctl -w /etc/passwd -p wa -k identity
auditctl -w /etc/sudoers -p wa -k privilege
auditctl -a always,exit -F arch=b64 -S execve -F euid=0 -k root_exec
ausearch -k identity -ts today
aureport --summary; aureport --auth --failed
service auditd restart

• Rules live in /etc/audit/rules.d/*.rules (generated to /etc/audit/audit.rules); manage them as code.
• Baseline rule categories: identity files, privilege escalation, kernel module loads, network config changes, time changes, and execution by privileged users.
• -F arch=b64 plus a b32 rule on x86_64 for complete coverage; syscall auditing of execve is high-volume, so scope it (e.g., euid=0) and size the log partition.
• -e 2 makes the audit configuration immutable until reboot; use on hardened hosts and plan rule changes around reboots.
• Audit only what you will read: forward to a SIEM, alert on --auth --failed and identity-key events. Unread audit logs are cost without control.
• auditd can be throttled under event storms (-r rate limit, backlog warnings); a drops-in-audit message means evidence is missing.
• Consider auditd alongside sysmon-for-Linux or eBPF-based monitoring for richer process telemetry.

Firewall Defaults

| Layer | Default | Tooling | |---|---|---| | Host (Linux) | Deny inbound, allow outbound, drop invalid | nftables, firewalld, ufw | | Host (macOS) | Application firewall on; stealth mode | socketfilterfw, pf for advanced | | Host (Windows) | Domain/Private deny inbound; Public strict | Defender Firewall, GPO | | Cloud | Security group/NACL deny by default | Terraform/console; review drift | | Container | Publish only required ports | Runtime network config |

nft list ruleset | tee /etc/nftables.conf    # review before persisting
firewall-cmd --state; firewall-cmd --list-all
ufw status verbose
/usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate

• Host firewall and cloud security group must agree; the effective policy is the intersection. Test from outside, not just localhost.
• Egress filtering is underused: servers rarely need unrestricted outbound. Allow DNS, NTP, package mirrors, and known peers; default-deny the rest where feasible.
• Log denied traffic deliberately; logging everything floods disks and hides signals.
• Rate-limit SSH/ICMP at the firewall edge; do not rely on fail2ban alone.
• Review rules as code: diff after changes, remove stale allows for decommissioned services and IPs.

CIS-Style Baselines and Evidence

# OpenSCAP (RHEL family)
oscap xccdf eval --profile cis --results results.xml --report report.html \
  /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml
# Lynis
lynis audit system
# Docker Bench (container host)
docker-bench-security

• CIS Benchmarks are prescriptive but not free: each profile level has operational tradeoffs. Choose Level 1 as baseline, evaluate Level 2 items individually.
• Automated scanners produce false positives and drift; triage findings and keep a deviation register with owner, reason, and expiry.
• Score is not security: 100% on a scanner with unpatched services is worse than 80% with current patches and monitoring.
• Re-scan after changes and after upgrades; baselines are living configuration, not one-time projects.
• Pair technical controls with identity controls: MFA, privileged access management, break-glass accounts with rotation, and session recording for high-risk access.
• Managed fleets: MDM/GPO/DSC/Ansible ensure drift correction; scanners only measure.

Least Privilege in Practice

• Accounts — one identity per human, no shared logins; service accounts with non-login shells; disable unused accounts; review group membership quarterly.
• sudoers — visudo -c validation; rules by command path and arguments; no NOPASSWD: ALL except documented break-glass; Defaults use_pty and log_output.
• Capabilities — prefer CAP_NET_BIND_SERVICE over running as root; check with getcap -r / 2>/dev/null and remove unnecessary setuid (find / -perm -4000).
• File permissions — umask 027, 600 for keys and secrets, 640 for shared config; audit world-writable files (find / -xdev -perm -0002 -type f).
• Secrets — managers (Vault, SOPS, cloud secret stores) over files; never in Git, shell history, or process arguments. Rotate on staff changes and suspected exposure.
• Service isolation — dedicated users, systemd sandboxing (ProtectSystem=strict, PrivateTmp=, NoNewPrivileges=, SystemCallFilter=), AppArmor/SELinux confinement.
• Network — segmentation and egress rules limit lateral movement; default-deny between tiers, not just at the perimeter.
• Kernel — enable kernel.dmesg_restrict=1, kernel.kptr_restrict=2, kernel.yama.ptrace_scope=1 where compatible; consider lockdown=confidentiality on high-value hosts.
• Backups — immutable/offline copies defeat ransomware and credential theft; restore credentials are separate from production credentials.
• Integrity — AIDE/Tripwire for file integrity, rpm -Va/debsums for package verification, and boot integrity (Secure Boot, measured boot) where required.

Incident Triage Checklist (Host)

1. Preserve evidence first: memory (if warranted), disk image or snapshot, and logs before changes. Note time source and timezone.
2. Identify the process tree and network connections: ps auxf, ss -tupn, lsof -i.
3. Check persistence: cron/timers, systemd units, launchd items, scheduled tasks, SSH keys, shell rc files, and package changes.
4. Check accounts and auth: last, lastlog, sudo/audit logs, new UIDs, group changes.
5. Check files: recently modified binaries (rpm -Va, find -mtime), deleted-but-open files (lsof +L1), and unexpected listening ports.
6. Isolate at the network layer (not by wiping), keep the host powered for forensics if policy requires, and notify per the incident plan.
7. Eradicate by rebuilding from a known-good image when integrity is uncertain; treat credential rotation as mandatory after any confirmed compromise.

Anti-Patterns

• Disabling SELinux/AppArmor, lowering firewall rules, or granting NOPASSWD: ALL to make an error disappear.
• Changing the SSH port and calling it security while leaving passwords enabled.
• Treating an automated CIS score as security; ignoring deviations.
• Storing long-lived credentials in environment variables or files readable by others.
• Patching only when prompted by an incident.
• Auditing everything and reading nothing.
• Assuming internal networks are trusted; flat networks amplify any foothold.
• Leaving default credentials on management interfaces (BMC/IPMI, databases, routers).
• Rebuilding a compromised host in place rather than from a known-good image.

Checklist

• [ ] Threat model and data classification documented for the host/role.
• [ ] OS and applications on supported versions; EOL tracked; patch cadence defined.
• [ ] Automatic security updates enabled with monitoring and restart policy.
• [ ] SSH: keys only, root login disabled, restricted groups, MFA where required.
• [ ] Firewall default deny with explicit allows; egress considered; rules persisted.
• [ ] SELinux/AppArmor enforcing, exceptions documented.
• [ ] auditd rules in place, forwarded, and reviewed; log capacity sized.
• [ ] Least privilege: sudo/groups review, capabilities, file permissions, secrets.
• [ ] Failure monitoring and integrity checks deliver alerts someone acts on.
• [ ] Backups immutable/offline with restore tests and separate credentials.
• [ ] Baseline deviations recorded with owner and expiry; incidents have a runbook.

Related: Linux administration for units and privileges, networking for firewall and tunnel configuration, containers for isolation boundaries.