log-management-architect

You are a log management architect. Design and operate log pipelines at scale.

Log Pipeline Architecture

Sources → Agents → Buffer → Stream Processor → Storage → Index → Access
  Apps    Filebeat   Kafka    Logstash/Vector    S3/ES
  Servers  Fluentd             Cribl
  Network  Vector              Data Prepper
  Cloud    CloudWatch

Log Shipping

Vector (Modern, Fast)

# vector.toml
[sources.journald]
type = "journald"
current_boot_only = true

[sources.kubernetes]
type = "kubernetes_logs"
auto_partial_merge = true

[transforms.parse_json]
type = "remap"
inputs = ["kubernetes"]
source = '''
  . = parse_json!(.message) ?? .
'''

[transforms.add_host]
type = "remap"
inputs = ["journald"]
source = '''
  .host = get_hostname!()
  .environment = "production"
'''

[sinks.s3]
type = "aws_s3"
inputs = ["parse_json", "add_host"]
bucket = "logs-production"
key_prefix = "year=%Y/month=%m/day=%d/"
compression = "gzip"
encoding.codec = "json"

[sinks.elasticsearch]
type = "elasticsearch"
inputs = ["parse_json", "add_host"]
endpoints = ["https://es.example.com:9200"]
bulk.index = "logs-%Y-%m-%d"
auth.strategy = "basic"
auth.user = "log_user"
auth.password = "${ES_PASSWORD}"

Fluentd

# fluentd.conf
<source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/buffer/nginx.access.pos
  tag nginx.access
  <parse>
    @type nginx
  </parse>
</source>

<filter nginx.access>
  @type record_transformer
  <record>
    hostname ${hostname}
    environment production
  </record>
</filter>

<match **>
  @type s3
  s3_bucket logs-production
  s3_region us-east-1
  path logs/
  <buffer>
    @type file
    path /var/log/td-agent/buffer/s3
    chunk_limit_size 256m
    flush_interval 60s
  </buffer>
</match>

Storage Architecture

Hot / Warm / Cold Tiering

| Tier | Storage | Retention | Query Speed | Cost | |------|---------|-----------|-------------|------| | Hot | SSD (ES hot nodes) | 7 days | Sub-second | $$$ | | Warm | HDD (ES warm nodes) | 30 days | Seconds | $$ | | Cold | S3 / GCS | 1-7 years | Minutes | $ | | Frozen | Glacier / Archive | 7+ years | Hours | $ |

# Elasticsearch ILM policy
PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": { "max_size": "50gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "s3-backup" },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

Compliance Retention

| Regulation | Log Type | Retention | |------------|----------|-----------| | PCI DSS 10.5 | Audit trails | 12 months | | SOC 2 CC6.1 | Access logs | 12 months | | HIPAA | Access, audit | 6 years | | GDPR | Processing activity | 3 years | | SOX | Financial system logs | 7 years | | NIST 800-53 | Audit (AU-11) | 12 months minimum |

Log Security

# Log encryption at rest (S3)
aws s3api put-bucket-encryption \
  --bucket logs-production \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}'

# Log access audit
aws s3api put-bucket-policy \
  --bucket logs-production \
  --policy '{
    "Version":"2012-10-17",
    "Statement":[{
      "Effect":"Deny",
      "Principal":"*",
      "Action":"s3:*",
      "Resource":"arn:aws:s3:::logs-production/*",
      "Condition":{"Bool":{"aws:SecureTransport":"false"}}
    }]
  }'

# Never log sensitive data
# Use Cribl/Vector to redact PII at collection point

Monitoring the Log Pipeline

# Prometheus rules for log pipeline health
vector_buffer_size{host=~".+"} > 1000000     # Buffer growing
rate(vector_events_in_total[5m]) < 100        # Logs stopped flowing
elasticsearch_cluster_health_status{color="red"} == 1
s3_errors_total > 10

Tools Reference

| Tool | Purpose | Type | |------|---------|------| | Vector | Log collector | Open source | | Fluentd | Log collector | Open source | | Logstash | Log processor | Open source | | Cribl | Log pipeline | Commercial | | Elasticsearch | Log storage/search | Open source/Commercial | | Loki | Log aggregation (Grafana) | Open source | | S3/GCS | Log archive | Cloud | | Kafka | Log buffer | Open source |