Backups, and proving they work

Your real photographs go on the server in chapter 15. This chapter makes that safe.

Separate three kinds of protection #

Redundancy keeps a service running after one member disk fails. Deletion, corruption and ransomware are copied to every mirror member.

A snapshot preserves an earlier filesystem view, usually on the same machine. A block snapshot of a running database is crash-consistent unless the database is quiesced; use the application's dump tool for application consistency.

A backup is a restorable copy in a different failure domain. An off-machine copy survives losing the server. Ransomware survival additionally requires append-only access or object lock.

Include data by recovery value #

Back up the whole Immich upload root, then exclude only documented generated data. This retains originals in upload/, library/ and profile/ across both default and Storage Template layouts.

/srv/homeserver/data/photos/thumbs
/srv/homeserver/data/photos/encoded-video
**/*.tmp

The live PostgreSQL directory is not in the backup selection. A running database changes while files are copied; create a consistent logical dump and back up that dump instead.

Configure one off-machine repository #

restic encrypts content locally, deduplicates it and supports SFTP and S3-compatible storage. A local path is useful for a disposable test, but it does not satisfy this chapter. Choose a repository on another machine or at a storage provider and record which failures it survives.

Add two values to secrets/restic.env on the laptop:

RESTIC_REPOSITORY=<backend-and-off-machine-repository>
RESTIC_PASSWORD=<long-random-repository-password>

Add the explicit mappings to chapter 9's render-secrets.sh, reinstall the script on the server, and render:

render secrets/restic.env '["RESTIC_REPOSITORY"]' restic_repository
render secrets/restic.env '["RESTIC_PASSWORD"]' restic_password

$ sudo /usr/local/sbin/render-secrets

The repository password decrypts backup data. The age key decrypts committed configuration. Losing either is serious for a different reason, so keep independent off-server recovery copies of both.

Configure the repository's transport credential separately. Prefer an append-only key or a rest-server --append-only account for the home server. Verify the backend's current permissions rather than assuming that the word “backup” prevents deletion.

Install a small root-only wrapper so interactive commands use the same files as the scheduled job:

#!/usr/bin/env bash
set -Eeuo pipefail
export RESTIC_REPOSITORY_FILE=/etc/homeserver/secrets/restic_repository
export RESTIC_PASSWORD_FILE=/etc/homeserver/secrets/restic_password
exec /usr/bin/restic "$@"

$ sudo install -o root -g root -m 0755 homeserver-restic /usr/local/sbin/homeserver-restic
$ sudo /usr/local/sbin/homeserver-restic init
$ sudo /usr/local/sbin/homeserver-restic snapshots

Record the repository identifier and snapshot list from your own backend rather than copying sample output.

Create an atomic backup job #

The job below runs as root, creates the dump directory before redirecting output, validates the temporary dump, and replaces the last good dump only after success. It writes the backup-success metric only after restic succeeds.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

compose=/home/admin/homeserver/immich/compose.yaml
dump_dir=/srv/homeserver/appdata/dumps
metric_dir=/var/lib/node_exporter/textfile_collector
dump_tmp=
metric_tmp=
cleanup() {
  [[ -z ${dump_tmp:-} ]] || rm -f "$dump_tmp"
  [[ -z ${metric_tmp:-} ]] || rm -f "$metric_tmp"
}
trap cleanup EXIT

install -d -o root -g root -m 0700 "$dump_dir"
dump_tmp=$(mktemp "$dump_dir/.immich.dump.XXXXXX")
docker compose -f "$compose" exec -T database \
  pg_dump -U postgres -Fc immich >"$dump_tmp"
test -s "$dump_tmp"
docker compose -f "$compose" exec -T database \
  pg_restore --list <"$dump_tmp" >/dev/null
mv -f "$dump_tmp" "$dump_dir/immich.dump"
dump_tmp=

/usr/local/sbin/homeserver-restic backup \
  /srv/homeserver/data \
  /srv/homeserver/appdata/dumps \
  --exclude-file /etc/homeserver/backup-excludes \
  --tag scheduled

install -d -o root -g root -m 0755 "$metric_dir"
metric_tmp=$(mktemp "$metric_dir/.backup.XXXXXX")
printf '%s\n' \
  '# HELP homeserver_backup_last_success_seconds Unix time of the last successful backup.' \
  '# TYPE homeserver_backup_last_success_seconds gauge' \
  "homeserver_backup_last_success_seconds $(date +%s)" >"$metric_tmp"
chmod 0644 "$metric_tmp"
mv -f "$metric_tmp" "$metric_dir/homeserver_backup.prom"
metric_tmp=

Install it only after reviewing the resolved admin path:

$ sudo install -o root -g root -m 0755 homeserver-backup /usr/local/sbin/homeserver-backup

The source server does not run forget --prune. Giving this job deletion rights would let a compromise erase the remote history.

Schedule and inspect it with systemd #

[Unit]
Description=Home server backup
Wants=network-online.target
After=network-online.target docker.service

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/sbin/homeserver-backup

[Unit]
Description=Run the home server backup daily

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target

$ sudo systemd-analyze verify \
    /etc/systemd/system/homeserver-backup.service \
    /etc/systemd/system/homeserver-backup.timer
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now homeserver-backup.timer
$ sudo systemctl start homeserver-backup.service
$ sudo systemctl status homeserver-backup.service
$ systemctl list-timers homeserver-backup.timer
$ sudo journalctl -u homeserver-backup.service -n 50 --no-pager

Require a successful manual run before trusting the timer. Until chapter 14 alerts on the metric, check the timer and journal regularly; a failed unit is visible in systemctl --failed but does not notify you by itself.

Keep deletion authority elsewhere #

Configure a separate, secured laptop or maintenance host with a deletion-capable repository credential. Run retention there, not on the home server:

$ restic forget --keep-within 1y --prune

For append-only repositories, restic recommends time-window retention because count-based rules can be manipulated by forged snapshot timestamps. Review the snapshots selected by forget before permitting deletion, and record the maintenance cadence. Provider object lock is stronger when available because neither credential can delete locked objects early.

Check stored data, not only job exit status #

$ sudo /usr/local/sbin/homeserver-restic check
$ sudo /usr/local/sbin/homeserver-restic check --read-data-subset=1/12

Run a different deterministic fraction from 1/12 through 12/12 each month, or run a complete annual --read-data check when transfer cost permits. Repeating a random 5% sample would cover only about 46% in expectation after twelve runs, so it does not justify a full-coverage claim.

Restore files and a database on purpose #

Run from the server and keep the Compose path explicit:

$ drill=$(mktemp -d /var/tmp/homeserver-drill.XXXXXX)
$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml stop
$ sudo mv /srv/homeserver/data/photos /srv/homeserver/data/photos.gone
$ sudo /usr/local/sbin/homeserver-restic restore latest --target "$drill"
$ sudo find "$drill/srv/homeserver/data/photos" -type f | sort | head

The restored tree contains the full original path. Require originals under upload/ or library/ and profile data when present; require the excluded thumbs/ and encoded-video/ directories to be absent.

Put the files back, start Immich and compare each irreplaceable directory that existed before the drill:

$ sudo cp -a "$drill/srv/homeserver/data/photos" /srv/homeserver/data/
$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml up -d
$ for part in upload library profile; do \
    test ! -d "/srv/homeserver/data/photos.gone/$part" || \
      sudo rsync -aHAXnci "/srv/homeserver/data/photos.gone/$part/" \
        "/srv/homeserver/data/photos/$part/"; \
  done

No output from these checksum dry-runs means the irreplaceable content and metadata selected by the rsync flags match. Generated directories were deliberately excluded and may be regenerated.

Restore the dump into a scratch database. The redirection must occur in a root shell because the restored dump is root-readable; the dump reaches pg_restore through stdin rather than a nonexistent container path:

$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml \
    exec -T database createdb -U postgres restore_test
$ sudo bash -c 'docker compose -f /home/admin/homeserver/immich/compose.yaml \
    exec -T database pg_restore -U postgres -d restore_test < "$1"' _ \
    "$drill/srv/homeserver/appdata/dumps/immich.dump"
$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml \
    exec -T database psql -U postgres -d restore_test -c '\dt'

Choose one meaningful table from your installed Immich version and compare its row count between the live and scratch databases. A successful restore plus plausible application data passes this drill; merely listing tables is not enough. Then remove the scratch database:

$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml \
    exec -T database dropdb -U postgres restore_test

Remove /srv/homeserver/data/photos.gone and the exact mktemp directory only after the application works and comparisons pass:

$ sudo rm -rf -- /srv/homeserver/data/photos.gone
$ case "$drill" in /var/tmp/homeserver-drill.*) sudo rm -rf -- "$drill";; *) false;; esac

The guarded case refuses a broad or unexpected path. Record the commands, elapsed time, missing content and any permissions you had to fix.

Repeat the drill twice a year #

Applications add paths, schemas change and credentials expire. Repeat both file and database restores twice a year and after adding any stateful service. Chapter 13 extends this job and drill for Authentik before central login becomes a dependency.