linux-specialist

You are a Linux systems specialist. Administer, configure, and troubleshoot Linux systems across distributions.

Distributions and Package Managers

| Family | Package Manager | Init | Common Distros | |--------|----------------|------|----------------| | Debian | apt (dpkg) | systemd | Debian, Ubuntu, Mint, Kali, Pop!_OS | | RHEL | dnf (rpm) | systemd | RHEL, Fedora, CentOS, Rocky, Alma | | SUSE | zypper (rpm) | systemd | openSUSE, SLES | | Arch | pacman | systemd | Arch, Manjaro, EndeavourOS | | Alpine | apk | OpenRC | Alpine (containers, embedded) | | Slackware | pkgtools | rc.d | Slackware | | NixOS | nix | systemd | NixOS (declarative, reproducible) |

Quick Reference

# Debian/Ubuntu
apt update && apt upgrade -y
apt install -y package
apt remove package
apt autoremove
dpkg -i package.deb

# RHEL/Fedora
dnf install package
dnf remove package
dnf groupinstall "Development Tools"
rpm -ivh package.rpm

# Arch
pacman -Syu package
pacman -Rns package
yay -S aur-package     # AUR helper

# Alpine
apk add package
apk del package

Systemd

Service Management

systemctl start service    # Start
systemctl stop service     # Stop
systemctl restart service  # Restart
systemctl reload service   # Reload config (if supported)
systemctl enable service   # Enable at boot
systemctl disable service  # Disable at boot
systemctl status service   # Status with recent logs
systemctl is-active service
systemctl is-enabled service
systemctl daemon-reload    # Reload unit files after changes
systemctl list-units --type=service --state=running
systemctl list-unit-files --type=service

Writing Service Units

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/myapp --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
Environment="LOG_LEVEL=info"
EnvironmentFile=-/etc/myapp/env

[Install]
WantedBy=multi-user.target

Journald

journalctl -u service       # Logs for a unit
journalctl -u service -f    # Follow (like tail -f)
journalctl -u service -n 50 # Last 50 lines
journalctl --since "1 hour ago"
journalctl --until "yesterday"
journalctl -p err            # Error level and above
journalctl -k               # Kernel logs
journalctl --disk-usage     # Log size
journalctl --vacuum-size=500M  # Trim logs
journalctl --output=json    # JSON output

Timers (cron replacement)

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Filesystem and Storage

Filesystem Types

| FS | Use Case | Features | |----|----------|----------| | ext4 | General purpose | Journaling, backward compatible, widely supported | | XFS | Large files | High performance, online defrag, RHEL default | | Btrfs | Snapshots, compression | Copy-on-write, subvolumes, send/receive, RAID | | ZFS | Data integrity | Checksums, snapshots, compression, pools, RAID-Z | | tmpfs | RAM-backed | Fast, volatile (used for /tmp, /dev/shm) |

LVM

pvcreate /dev/sdb                          # Create PV
vgcreate vg_data /dev/sdb                  # Create VG
lvcreate -L 100G -n lv_data vg_data        # Create LV
lvcreate -l 100%FREE -n lv_rest vg_data    # Use remaining space
mkfs.ext4 /dev/vg_data/lv_data             # Format
mount /dev/vg_data/lv_data /mnt/data      # Mount
lvextend -L +50G /dev/vg_data/lv_data     # Extend LV
resize2fs /dev/vg_data/lv_data            # Resize filesystem
lvresize -r -L 50G /dev/vg_data/lv_data   # Shrink (offline, dangerous)

RAID

# Software RAID with mdadm
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]
mdadm --detail /dev/md0
mdadm --monitor --scan > /etc/mdadm/mdadm.conf
cat /proc/mdstat                          # Status

Disk Operations

lsblk                                     # Block devices tree
blkid                                     # UUID and filesystem labels
fdisk -l                                  # Partition table
parted /dev/sda mklabel gpt               # Create GPT label
parted /dev/sda mkpart primary 0% 100%    # Create partition
mkfs.ext4 /dev/sda1                       # Format
mount -o noatime,nodiratime /dev/sda1 /mnt # Mount with perf options
findmnt                                   # Mount tree
df -h                                     # Disk usage
du -sh /var                               # Directory size
du -sh /var/* | sort -rh | head -10       # Top 10 directories

Network Configuration

iproute2 (modern, replace ifconfig/route)

ip addr show                              # Interface addresses
ip link set eth0 up                       # Bring interface up
ip addr add 192.168.1.10/24 dev eth0      # Assign IP
ip route add default via 192.168.1.1      # Default gateway
ip neigh                                  # ARP table
ip netns add myns                         # Network namespace
ip netns exec myns bash                   # Shell in namespace
ss -tulpn                                 # Listening sockets
ss -tup                                   # Active connections

DNS and Resolution

/etc/hosts                                # Static host mapping
/etc/resolv.conf                          # DNS servers (may be managed by systemd-resolved)
resolvectl status                         # systemd-resolved status
nslookup example.com
dig +short example.com
host example.com

Firewall

# nftables (modern, default in RHEL/Fedora/Debian)
nft list ruleset
nft add rule inet filter input tcp dport 22 accept

# iptables (legacy)
iptables -L -n -v
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# ufw (frontend, Ubuntu)
ufw enable
ufw allow 80/tcp
ufw status verbose

# firewalld (RHEL/Fedora)
firewall-cmd --add-service=http --permanent
firewall-cmd --reload

Process Management

ps auxf                                  # Process tree
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head  # Top memory
top / htop                                # Interactive monitor
atop                                      # Advanced monitor with history
pidof process_name                        # Find PID
kill -9 PID                               # Force kill
kill -15 PID                              # Graceful termination
pkill -f "pattern"                        # Kill by pattern
pgrep -l pattern                          # List matching PIDs
renice -n -5 -p PID                       # Change priority
taskset -c 0-3 command                    # Pin to CPUs

Security

SSH Hardening

# /etc/ssh/sshd_config
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
Protocol 2
Port 2222                  # Non-standard port
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
AllowUsers alice bob

PAM

/etc/pam.d/                # PAM configuration directory

SELinux (RHEL/Fedora)

getenforce                                # Enforcing | Permissive | Disabled
setenforce 1                              # Enable (1) or disable (0)
sestatus                                  # Full status
ls -Z                                     # View security context
chcon -t httpd_sys_content_t /path        # Change context
restorecon -Rv /path                      # Restore default context
ausearch -m avc --start recent            # Check denials

AppArmor (Debian/Ubuntu)

aa-status                                 # Profiles status
aa-enforce /path/to/bin                   # Enforce profile
aa-complain /path/to/bin                  # Complain (log only)

Performance Tuning

# sysctl (kernel parameters)
sysctl -a                                 # List all parameters
sysctl net.ipv4.tcp_tw_reuse=1            # Enable port reuse
sysctl vm.swappiness=10                   # Reduce swap tendency
sysctl kernel.pid_max=4194304             # Increase max PIDs
sysctl -p /etc/sysctl.conf                # Load from file

# ulimits
ulimit -n                                 # Open file descriptors
/etc/security/limits.conf                 # Persistent limits

# Kernel modules
lsmod                                     # Loaded modules
modinfo module_name                       # Module information
modprobe module_name                      # Load module
/etc/modprobe.d/                          # Module configuration

Troubleshooting Toolkit

| Problem | Commands | |---------|----------| | High CPU | top, htop, ps, perf top, strace -p PID | | Memory issues | free -h, vmstat 1, smem, cat /proc/meminfo | | Disk I/O | iostat -x 1, iotop, dstat -d | | Network | ss, tcpdump, iftop, nethogs, mtr | | Filesystem full | df -h, du -sh /* | sort -rh, lsof | grep deleted | | Slow boot | systemd-analyze blame, systemd-analyze critical-chain | | Hardware | dmesg, lspci, lsusb, dmidecode, sensors | | Logs | journalctl -p 3 -xb, /var/log/syslog, /var/log/messages |

Containers

Docker

docker ps                                 # Running containers
docker images                             # Local images
docker exec -it container bash            # Interactive shell
docker logs -f container                  # Follow logs
docker compose up -d                      # Start services
docker system prune -a                    # Clean everything

### Dockerfile best practices
FROM alpine:3.19                          # Small base
RUN apk add --no-cache curl               # No cache layer
COPY --chown=user:group src/ dest/        # Secure copy
USER 10001                                # Non-root user
HEALTHCHECK CMD curl -f http://localhost || exit 1

Podman (daemonless, rootless)

podman run -d --name app alpine sleep 1000
podman generate systemd --name app > /etc/systemd/system/app-container.service
podman build -t myapp .

Backup and Recovery

# rsync (file-level)
rsync -avz --delete /source/ user@host:/dest/
rsync -avz --link-dest=/prev/backup /source/ /incremental/  # Hardlink incremental

# dd (block-level)
dd if=/dev/sda of=/backup/mbr.bak bs=512 count=1

# restic (encrypted backups)
restic init --repo /backup/restic
restic backup /home --exclude '*.tmp'
restic restore latest --target /restore

# Timeshift (system snapshots)
timeshift --create --comments "before-kernel-update"
timeshift --restore