Living with it / Chapter 14

Living with it Chapter 14 120 min

Knowing it is alive

Deploy a small private monitoring stack, alert on useful symptoms, and test the complete notification path.

By the end of this chapter you should be able to

  • Say what metrics, logs and alerts are each good for
  • Deploy a bounded monitoring stack without Docker API access
  • Alert on service, disk, backup and reboot state
  • Break something on purpose and receive the resulting email

Right now a household member is the monitoring system. This chapter replaces that arrangement with a small number of checks you will act on.

Give each signal one job #

Metrics are numbers over time. They show trends and thresholds. Logs are detailed events. They explain what happened. Alerts evaluate metrics and notify a person.

Prometheus collects metrics and evaluates rules. node_exporter reports the host. blackbox_exporter makes an HTTP request. Alertmanager delivers notifications. Loki retains logs, Alloy sends Docker's JSON log files to it, and Grafana queries both stores.

The stack deliberately has no Docker socket. It still grants two narrow but powerful host views. Alloy can read Docker's log directory, whose files may contain names, URLs or tokens. node_exporter can read the host filesystem and see host process identifiers so its metrics describe the host rather than its container. Both mounts are read-only, the images are pinned, retention is short, and none of the interfaces is public.

Add proxy metrics #

Add these arguments to Traefik's command list and attach it to the new monitoring network when you merge the Compose fragments below:

Traefik additions
command:
  - --metrics.prometheus=true
  - --entrypoints.metrics.address=:8082
  - --metrics.prometheus.entrypoint=metrics
networks: [edge, edge-photos, monitoring]

networks:
  monitoring:
    external: true
    name: homeserver-monitoring

Do not publish port 8082. Prometheus reaches it on the Docker network.

Create the configuration #

Create ~/homeserver/monitoring/config. The probe below connects directly to Traefik but sends the real hostname for routing and certificate verification. It therefore checks Traefik, TLS and Immich without depending on container DNS for the household hostname.

config/blackbox.yml
modules:
  photos_origin:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200, 301, 302, 401]
      fail_if_not_ssl: true
      headers:
        Host: photos.example.com
      tls_config:
        server_name: photos.example.com
        ca_file: /etc/blackbox/home-server-ca.crt
config/prometheus.yml
global:
  scrape_interval: 30s

rule_files:
  - /etc/prometheus/rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: [alertmanager:9093]

scrape_configs:
  - job_name: node
    static_configs:
      - targets: [node-exporter:9100]

  - job_name: traefik
    static_configs:
      - targets: [traefik:8082]

  - job_name: photos-origin
    metrics_path: /probe
    params:
      module: [photos_origin]
    static_configs:
      - targets: [https://traefik:443/]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter:9115

The disk rule combines a trend with a hard floor. A trend alone cannot predict a sudden large write, and a floor alone warns late. The backup rule also treats a missing metric as failure; comparing an absent series with a number produces no alert.

config/rules.yml
groups:
  - name: homeserver
    rules:
      - alert: PhotosOriginDown
        expr: probe_success{job="photos-origin"} == 0
        for: 5m
        annotations:
          summary: "The local Immich origin path has failed for five minutes"

      - alert: ScrapeTargetMissing
        expr: up{job=~"node|traefik"} == 0
        for: 5m
        annotations:
          summary: "Prometheus cannot scrape {{ $labels.job }}"

      - alert: RootDiskWillFill
        expr: |
          predict_linear(node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"}[6h], 4*24*3600) < 0
          or
          node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"} < 10*1024*1024*1024
        for: 1h
        annotations:
          summary: "The root filesystem needs attention"

      - alert: BackupStale
        expr: |
          absent(homeserver_backup_last_success_seconds)
          or
          time() - homeserver_backup_last_success_seconds > 172800
        for: 15m
        annotations:
          summary: "No complete backup has succeeded for two days"

      - alert: RebootRequired
        expr: homeserver_reboot_required == 1
        for: 24h
        annotations:
          summary: "A security-update reboot has been pending for a day"

      - alert: HostMetricMissing
        expr: absent(homeserver_reboot_required)
        for: 30m
        annotations:
          summary: "The host-state metric has disappeared"

Confirm the filesystem label before trusting the disk expression:

$ df -h /srv

If /srv is a separate mounted filesystem, replace mountpoint="/" with its actual node_exporter label. Test every expression in Prometheus; an empty result is not a passing check.

Write the reboot metric alongside chapter 11's backup metric:

/usr/local/sbin/homeserver-host-metrics
#!/usr/bin/env bash
set -Eeuo pipefail
metric_dir=/var/lib/node_exporter/textfile_collector
install -d -o root -g root -m 0755 "$metric_dir"
tmp=$(mktemp "$metric_dir/.host.XXXXXX")
trap 'rm -f "$tmp"' EXIT
value=0
test ! -f /var/run/reboot-required || value=1
printf '%s\n' \
  '# HELP homeserver_reboot_required Whether the host requires a reboot.' \
  '# TYPE homeserver_reboot_required gauge' \
  "homeserver_reboot_required $value" >"$tmp"
chmod 0644 "$tmp"
mv -f "$tmp" "$metric_dir/homeserver_host.prom"
trap - EXIT

Install it and run it from a root systemd timer every fifteen minutes:

$ sudo install -o root -g root -m 0755 homeserver-host-metrics \
    /usr/local/sbin/homeserver-host-metrics
/etc/systemd/system/homeserver-host-metrics.service
[Unit]
Description=Export home server host-state metrics

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/sbin/homeserver-host-metrics
/etc/systemd/system/homeserver-host-metrics.timer
[Unit]
Description=Refresh home server host-state metrics

[Timer]
OnBootSec=2m
OnUnitActiveSec=15m

[Install]
WantedBy=timers.target

$ sudo systemd-analyze verify \
    /etc/systemd/system/homeserver-host-metrics.service \
    /etc/systemd/system/homeserver-host-metrics.timer
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now homeserver-host-metrics.timer
$ sudo systemctl start homeserver-host-metrics.service
$ systemctl list-timers homeserver-host-metrics.timer

Configure email without putting its password in Git. Add SMTP_PASSWORD and GRAFANA_ADMIN_PASSWORD to secrets/monitoring.env, then add these mappings to chapter 9's renderer:

render secrets/monitoring.env '["SMTP_PASSWORD"]' smtp_password
render secrets/monitoring.env '["GRAFANA_ADMIN_PASSWORD"]' grafana_admin_password
config/alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.net:587'
  smtp_from: 'server@example.com'
  smtp_auth_username: 'server@example.com'
  smtp_auth_password_file: /run/secrets/smtp_password

route:
  receiver: email
  group_by: [alertname]
  group_wait: 30s
  repeat_interval: 4h

receivers:
  - name: email
    email_configs:
      - to: 'alerts@example.com'
        send_resolved: true

Set short, hard retention limits for logs:

config/loki.yml
auth_enabled: false
server:
  http_listen_port: 3100
common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  retention_period: 168h
compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  delete_request_store: filesystem
config/config.alloy
local.file_match "docker" {
  path_targets = [{
    __path__ = "/var/lib/docker/containers/*/*-json.log",
    job      = "docker",
  }]
}

loki.source.file "docker" {
  targets    = local.file_match.docker.targets
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint { url = "http://loki:3100/loki/api/v1/push" }
}

Deploy the private stack #

Pin every image to a reviewed digest as in chapter 7. Keep monitoring/compose.yaml and monitoring/config/ in the canonical laptop repository, commit the non-secret files, and repeat chapter 5's rsync deployment. The Prometheus and Loki volumes are disposable history. Export the one Grafana dashboard as JSON into the repository whenever you change it, so rebuilding Grafana does not depend on its named volume.

On the server, create one explicitly named ordinary network before starting either project, then redeploy Traefik with the addition above:

$ sudo docker network inspect homeserver-monitoring >/dev/null 2>&1 || \
    sudo docker network create homeserver-monitoring
$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml up -d traefik

The monitoring project attaches to the same external network:

~/homeserver/monitoring/compose.yaml
name: monitoring
services:
  prometheus:
    image: quay.io/prometheus/prometheus:<release>@sha256:<pin it>
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=15d
      - --storage.tsdb.retention.size=8GB
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./config/rules.yml:/etc/prometheus/rules.yml:ro
      - prometheus-data:/prometheus
    ports: ["127.0.0.1:9090:9090"]
    networks: [monitoring]
    restart: unless-stopped

  node-exporter:
    image: quay.io/prometheus/node-exporter:<release>@sha256:<pin it>
    command:
      - --path.rootfs=/host
      - --collector.textfile.directory=/textfiles
    pid: host
    volumes:
      - /:/host:ro,rslave
      - /var/lib/node_exporter/textfile_collector:/textfiles:ro
    networks: [monitoring]
    restart: unless-stopped

  blackbox-exporter:
    image: quay.io/prometheus/blackbox-exporter:<release>@sha256:<pin it>
    volumes:
      - ./config/blackbox.yml:/config/blackbox.yml:ro
      - /etc/homeserver/trust/home-server-ca.crt:/etc/blackbox/home-server-ca.crt:ro
    command: [--config.file=/config/blackbox.yml]
    networks: [monitoring]
    restart: unless-stopped

  alertmanager:
    image: quay.io/prometheus/alertmanager:<release>@sha256:<pin it>
    volumes: ["./config/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro"]
    secrets: [smtp_password]
    networks: [monitoring]
    restart: unless-stopped

  loki:
    image: docker.io/grafana/loki:<release>@sha256:<pin it>
    command: ["-config.file=/etc/loki/loki.yml"]
    volumes:
      - ./config/loki.yml:/etc/loki/loki.yml:ro
      - loki-data:/loki
    networks: [monitoring]
    restart: unless-stopped

  alloy:
    image: docker.io/grafana/alloy:<release>@sha256:<pin it>
    command: [run, /etc/alloy/config.alloy, --storage.path=/var/lib/alloy/data]
    volumes:
      - ./config/config.alloy:/etc/alloy/config.alloy:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - alloy-data:/var/lib/alloy/data
    networks: [monitoring]
    restart: unless-stopped

  grafana:
    image: docker.io/grafana/grafana:<release>@sha256:<pin it>
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
    secrets: [grafana_admin_password]
    volumes: [grafana-data:/var/lib/grafana]
    ports: ["127.0.0.1:3000:3000"]
    networks: [monitoring]
    restart: unless-stopped

networks:
  monitoring:
    external: true
    name: homeserver-monitoring

volumes:
  prometheus-data:
  loki-data:
  alloy-data:
  grafana-data:

secrets:
  smtp_password:
    file: /etc/homeserver/secrets/smtp_password
  grafana_admin_password:
    file: /etc/homeserver/secrets/grafana_admin_password

Then validate configurations and start:

$ cd /home/admin/homeserver/monitoring
$ sudo /usr/local/sbin/render-secrets
$ sudo docker compose config --quiet
$ sudo docker compose run --rm --entrypoint /bin/promtool prometheus \
    check config /etc/prometheus/prometheus.yml
$ sudo docker compose run --rm --entrypoint /bin/promtool prometheus \
    check rules /etc/prometheus/rules.yml
$ sudo docker compose run --rm --entrypoint /bin/amtool alertmanager \
    check-config /etc/alertmanager/alertmanager.yml
$ sudo docker compose up -d
$ sudo docker compose ps

Reach Grafana and Prometheus over the private SSH path rather than publishing them:

$ ssh -L 3000:127.0.0.1:3000 -L 9090:127.0.0.1:9090 admin@homeserver

Open http://127.0.0.1:3000 and add http://prometheus:9090 and http://loki:3100 as data sources. One dashboard is enough: disk free, memory, target state, request errors and time since the last backup.

Test the entire alert path #

Stop Immich using its actual project file, time the result, then restore it:

$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml stop immich-server
$ date

Watch the probe go from failed to pending to firing, receive the email, then run:

$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml up -d immich-server

Confirm the resolution email. Do not create a large file to test the disk rule; test the expression and use a temporary threshold in a copy of the rule instead.

Know what this stack cannot observe

The probe runs on the server. It detects failure of the local proxy, certificate and application path, but not loss of the host, electricity, router or home internet connection. Detecting those failures requires a check in another failure domain, such as a separately operated dead-man service watching a heartbeat. Give that service no administrative access.

Metrics or rules are missing

Find the first broken link before changing the alert

Paste this into a new agent session. It carries everything the agent needs to know about where you are, and asks it to walk you through the problem rather than fix it for you.

Prometheus shows an empty expression or a target is down. I want to inspect target labels, mounts, network reachability and rule loading one step at a time.

An alert fired but no email arrived

Trace notification delivery without exposing the SMTP password

Paste this into a new agent session. It carries everything the agent needs to know about where you are, and asks it to walk you through the problem rather than fix it for you.

Prometheus shows a firing alert. I want to check its Alertmanager delivery, routing and SMTP error logs without printing secret files.

Done when

  • All images are pinned and every configuration validator passes
  • Prometheus scrapes node_exporter and Traefik with bounded retention
  • The origin probe checks TLS, proxy routing and Immich
  • Backup-stale alerts when the metric is old or absent
  • Disk and reboot rules return the expected series
  • Loki retention is one week and the log mount is read-only
  • You recorded and accepted node_exporter's read-only root mount and host PID view
  • Grafana and Prometheus listen on loopback and are reached over private SSH
  • Stopping Immich produced an email and restarting it produced a resolution
  • The measured detection time is recorded
  • An independent check exists if host, power or internet loss must be detected

What you picked up

  • Metrics show trends, logs explain events, and alerts request action.
  • A complete deployment needs services, configuration, networks, secrets and validation.
  • Missing metrics need explicit absent() handling.
  • Same-host probing tests the origin path, not the host's failure domain.
  • Keep monitoring private, bound its storage and test notification delivery by causing a safe failure.

Settings

Your values

The book is written with placeholder names so it makes sense to everybody. Put your own in and every chapter, every command and every copy-paste prompt updates to match.

Nothing here is sent anywhere. It is saved in this browser, so it comes back next time. A different browser or a private window gets the placeholders again.

Live preview

$ ssh admin@192.168.1.20
$ sudo ufw allow from 192.168.1.0/24 to any port 22 proto tcp
$ sudo hostnamectl set-hostname homeserver
$ sudo timedatectl set-timezone Europe/Paris

Real commands from chapters 2, 3 and 4. They change as you type.

The account you log in as. Not root, and not necessarily the same name you use on your laptop.

Introduced in Chapter 2, Meet your server

The book's placeholder is admin

What the machine calls itself. You choose it, and it shows up in your shell prompt and your logs.

Introduced in Chapter 3, A safe front door

The book's placeholder is homeserver

The IP address your server has on your home network, from ip -brief addr.

Introduced in Chapter 2, Meet your server

The book's placeholder is 192.168.1.20

The address range and prefix shown by ip route or ip -brief addr, written in CIDR form. Copy the real prefix; do not guess /24.

Introduced in Chapter 2, Meet your server

The book's placeholder is 192.168.1.0/24

The address traffic goes to on its way out of your house, from ip route.

Introduced in Chapter 2, Meet your server

The book's placeholder is 192.168.1.1

In Region/City form, or Etc/UTC if you would rather read logs in UTC.

Introduced in Chapter 3, A safe front door

The book's placeholder is Europe/Paris

A registered name you control. Chapter 8 uses it for the LAN route; chapter 12 uses a separate private Tailscale name remotely.

Introduced in Chapter 8, One door, many rooms

The book's placeholder is example.com

The email identity allowed to administer the tagged server in your Tailscale policy.

Introduced in Chapter 10, Your own private network

The book's placeholder is you@example.com

The mailbox that should receive actionable home-server alerts.

Introduced in Chapter 14, Knowing it is alive

The book's placeholder is alerts@example.com

Once you save, the prose and the commands read with your names, the copy buttons copy your values, and the copy-paste prompts describe your machine accurately. That last one matters: an assistant told your network is 192.168.1.0/24 when it is not will send you chasing the wrong thing.

Anything you leave empty keeps the book's placeholder.