Keeping secrets

Chapter 7 left a database password in a plain file. This chapter fixes it, starting with the part nearly every tutorial gets wrong.

Where a secret in an environment variable can be read #

Run two disposable containers with the same dummy password, one passed as an environment variable and one as a file:

$ printf '%s' 'super-secret-value' | sudo tee /var/tmp/leaklab-password >/dev/null
$ sudo chmod 0600 /var/tmp/leaklab-password
$ sudo docker run -d --name leaklab-viaenv \
    -e DB_PASSWORD=super-secret-value alpine:3.22 sleep 300
$ sudo docker run -d --name leaklab-viafile \
    -e DB_PASSWORD_FILE=/run/secrets/db_password \
    -v /var/tmp/leaklab-password:/run/secrets/db_password:ro \
    alpine:3.22 sleep 300

Now inspect both forms.

$ sudo docker inspect leaklab-viaenv --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -i pass

Require your output to contain the complete dummy DB_PASSWORD value. That is documented container metadata, readable by anything that can talk to Docker.

The same question of the file-backed one:

$ sudo docker inspect leaklab-viafile --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -i pass

Require only DB_PASSWORD_FILE and its path, not the value.

That is one leak. There are more, and they are worth knowing individually because each has a different fix:

docker inspect shows it, as above. Anything with broad read access to the Docker container API can read environment variables and can also reach logs, process arguments and container file-export endpoints. Chapter 8 deliberately avoided Docker discovery for this reason.

docker compose config renders your file with variables substituted, which is a genuinely useful debugging command that prints your passwords to the terminal:

$ sudo docker compose config | grep -i password

Run this only while the known dummy value is configured, and confirm that Compose renders the value rather than merely its source expression.

/proc/<pid>/environ holds the environment of a running process, on the host, with no Docker command involved. Root can read any of them.

Everything downstream. A crash handler that dumps the environment into a log, a subprocess that inherits it, a support tool that prints configuration. Each turns one leak into a permanent one.

File-backed secrets #

Compose has a mechanism for this. Declare a secret pointing at a file, and it appears inside the container under /run/secrets/:

services:
  database:
    image: ghcr.io/immich-app/postgres:...
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/immich_db_password
    secrets: [immich_db_password]

  immich-server:
    image: ghcr.io/immich-app/immich-server:...
    environment:
      DB_PASSWORD_FILE: /run/secrets/immich_db_password
    secrets: [immich_db_password]

secrets:
  immich_db_password:
    file: /etc/homeserver/secrets/immich_db_password

Both services need it, and each uses its own variable name: the Postgres image knows POSTGRES_PASSWORD_FILE and the application knows DB_PASSWORD_FILE. Converting one and not the other is the most common way this goes wrong, and it fails as "the application cannot connect to the database".

Inside the container it is an ordinary file:

$ sudo docker exec leaklab-viafile sh -c 'cat /run/secrets/db_password; ls -l /run/secrets/'

The first line should be the dummy value; the listing shows the ownership and mode inherited from the host file. Record what your machine reports instead of assuming a particular container uid.

Remove every lab artifact now:

$ sudo docker rm -f leaklab-viaenv leaklab-viafile
$ sudo rm -f /var/tmp/leaklab-password

The _FILE suffix is a convention, not a standard, and it is not universally supported. The current PostgreSQL and Immich images support the variables shown here. For another image, check its documentation rather than inferring support from the suffix. An ignored variable can leave an application broken or, worse, running without the protection you expected.

Now the configuration itself #

File-backed secrets solve the runtime leak. They do not solve storage. The files have to exist, and plaintext files on disk keep your configuration out of git.

The trick is to encrypt the values, commit the ciphertext, and decrypt at deploy time.

SOPS encrypts the values in a structured file and leaves the keys readable, which is the property that makes it usable: a diff still shows you which setting changed. age is the encryption underneath, chosen here because it is one keypair and no ceremony.

Install both on the laptop, where you edit the canonical repository, and on the server, where deployment renders runtime files. Pin their release versions and checksums in the playbook or in a documented laptop setup file.

A keypair #

$ install -d -m 0700 ~/.config/sops/age
$ age-keygen -o ~/.config/sops/age/keys.txt
$ chmod 0600 ~/.config/sops/age/keys.txt
$ age-keygen -y ~/.config/sops/age/keys.txt

Record the age1... line printed by -y. It is the public key, or recipient: it encrypts, is not secret, and goes in your repository. Do not print or copy the AGE-SECRET-KEY-... line into a terminal transcript; it decrypts every file for that recipient.

One explicit transfer is easier to audit than an untracked second copy:

$ scp ~/.config/sops/age/keys.txt admin@192.168.1.20:/var/tmp/homeserver-age-key
$ ssh -t admin@192.168.1.20 \
    'sudo install -d -o root -g root -m 0700 /etc/homeserver/age && sudo install -o root -g root -m 0600 /var/tmp/homeserver-age-key /etc/homeserver/age/keys.txt && rm -f /var/tmp/homeserver-age-key'

Tell SOPS which key to use #

A .sops.yaml at the top of your repository means you never type the recipient again, and that anyone adding a file gets it encrypted the same way:

creation_rules:
  - path_regex: secrets/.*\.env$
    age: <your-age1-recipient>

This file is committed. It contains only the public half.

Encrypt something #

$ sops encrypt --input-type dotenv --output-type dotenv --in-place secrets/immich.env
$ sops filestatus secrets/immich.env
$ git diff -- secrets/immich.env

Require filestatus to report an encrypted file. Inspect the real diff: variable names and recipients remain readable, while values and the integrity MAC are encrypted. Never edit those metadata blocks by hand.

That file is safe to commit.

Prove decryption succeeds without printing the password:

$ sops decrypt --input-type dotenv --output-type dotenv secrets/immich.env >/dev/null

Change one value #

Edit on the laptop, where SOPS finds the protected key in its default age-key location:

$ sops set secrets/immich.env '["DB_PASSWORD"]' '"a-much-better-one"'
$ sops edit secrets/immich.env

sops edit opens your editor on the decrypted content, held in a temporary file it manages, and re-encrypts on save. sops set changes one value without an editor, which is what you want from a script.

sops edit uses a mode-0600 plaintext temporary file and removes it afterwards. It avoids a persistent plaintext copy, but /tmp may still be disk-backed. Use sops set for scripted single-value changes and keep the laptop encrypted if that temporary-file exposure matters to you.

Getting it to the containers at deploy time #

The pieces now connect. Encrypted file in git, private key on the server, file-backed secrets in compose. What joins them is a small script that runs before docker compose up.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

out=/etc/homeserver/secrets
install -d -o root -g root -m 0700 "$out"

export SOPS_AGE_KEY_FILE=/etc/homeserver/age/keys.txt
tmp=
cleanup() { [[ -z ${tmp:-} ]] || rm -f "$tmp"; }
trap cleanup EXIT

render() {
  local source=$1 expression=$2 target=$3
  tmp=$(mktemp "$out/.${target}.XXXXXX")
  sops decrypt --input-type dotenv --extract "$expression" "$source" >"$tmp"
  test -s "$tmp"
  chmod 0444 "$tmp"
  mv -f "$tmp" "$out/$target"
  tmp=
}

# One explicit mapping per runtime secret.
render secrets/immich.env '["DB_PASSWORD"]' immich_db_password

Three deliberate choices in there.

umask 077 at the top, so anything created is private by default rather than private if you remembered.

Each destination is replaced atomically. Decryption goes to a private temporary file in the destination directory; mv replaces the old secret only after the new one is non-empty and has the intended mode. The trap removes a partial file after an error.

Mode 0444, not 0400. The containers that read these files do not all run as root, and a root-only file is unreadable to an application running as uid 1000. What protects the secret is the directory: /etc/homeserver/secrets is 0700 and owned by root, so nothing on the host can list or open it except root. Inside the container, the only thing there is the application it belongs to.

Install the script as root and run it before Compose:

$ sudo install -o root -g root -m 0755 render-secrets.sh /usr/local/sbin/render-secrets
$ sudo /usr/local/sbin/render-secrets
$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml up -d

Remove the chapter 7 env_file: .env reference from both Immich services, validate Compose, and delete the server's plaintext .env only after the file-backed deployment works. The mapping list is the secret ledger. When a later chapter adds restic, Authentik or monitoring credentials, it adds an encrypted source key and one render line, then reruns the same script. A later chapter must not merely assume a /run/secrets/... file appeared.

Rotation #

Secrets differ in how hard they are to change. Find out which is which before an emergency.

Cheap. An API token for an external service. Generate a new one there, put it in the encrypted file, redeploy. Nothing else knows or cares.

Moderate. An application's own signing or session key. Changing it usually logs everybody out, which is disruptive rather than dangerous.

Expensive. A database password. The secret exists in two places: your file and the database's own user record. Change one and the application stops being able to connect, with an error that looks like the database is down. You have to change both, in the right order, and take the application down in between:

Stop the application, then let psql prompt without putting the value in shell history:

$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml stop immich-server
$ sudo docker compose -f /home/admin/homeserver/immich/compose.yaml \
    exec database psql -U postgres -d postgres -c '\password postgres'

Update the encrypted file on the laptop, deploy it, run sudo /usr/local/sbin/render-secrets, and start the application again. Practise this once on test data so both copies remain in step.

Effectively impossible without a rebuild. A key used to encrypt data at rest, where the stored data can only be read with the original. Some applications have one. Find out whether yours does before you need to know.

The one you cannot encrypt #

Everything is encrypted except the key that decrypts it, which sits in plaintext at /etc/homeserver/age/keys.txt. That is the design. Every secret-management system ends in one secret that exists in the clear, because something has to start without a human present. Cloud providers move it into hardware; you have moved it into one root-owned file on a machine you control.

What it buys you: your entire configuration can go in git and be published tomorrow. What it does not: anyone with root on the server, or an unencrypted disk in their hands, has everything. That is the physical-security dependency the lockout annex in chapter 3 described.