Make it repeatable

Without scrolling back, write down every change you have made to your server. Most people get about seven, in the wrong order, and miss the one that matters. Two chapters ago you had no way of knowing which those were.

If the machine dies, or you buy a bigger one, or you break something at 11pm, none of that is a procedure yet. It is you, remembering.

What we actually have #

A list of things that were true once:

• a hardening file in /etc/ssh/sshd_config.d/, whose name mattered for a reason you would have to remember
• a firewall policy, and one rule scoped to a subnet
• an upgrade configuration with a reboot setting you chose deliberately
• a hostname and a timezone

Four bullet points. The problem is not that this is a lot. The problem is that it lives in one place, that place is the server, and there is no way to ask "is it still like that?" other than checking each one by hand.

That question has a name. When a machine stops matching what you believe about it, because of an emergency fix nobody wrote down or a package upgrade that replaced a config file, that is drift. Drift is why people stop touching a working server.

Two ways to write down a setup #

The obvious approach is a script: the commands you ran, in order, in a file. It is a real improvement over memory, and if you stop reading here and write one, you are ahead of most home servers.

But run it twice and the trouble shows. The second run appends your SSH rule to the firewall again. It appends your key to authorized_keys again. Some commands fail because the thing already exists, and now you cannot tell a real failure from a harmless one. So you stop rerunning it. It becomes documentation nobody checks, and it stops being true.

The other approach is to describe the state you want rather than the steps to reach it. Say "this file should contain this" rather than "append this line", and "this rule should exist" rather than "run ufw allow". Something else works out what to change, and if nothing needs changing, it does nothing.

That property has a name too. An operation is idempotent when doing it twice has the same effect as doing it once.

Why Ansible #

Ansible is a tool for describing machine state in YAML and applying it over SSH. It has no agent to install on the server: it connects the same way you do, runs some Python, and disconnects.

That last point is why it fits here. You already have key-based SSH working. Ansible needs nothing else.

The honest cost of Ansible: YAML, which is a fussy format; a vocabulary of modules, plays, tasks and handlers; and documentation written for people managing five hundred machines when you have one. This chapter uses a small corner of it and ignores the rest.

Set it up on your laptop #

Ansible runs on the machine you sit at, not the one you are configuring. That machine is called the control node. Install it there:

$ sudo apt install ansible-core          # Debian, Ubuntu
$ brew install ansible                   # macOS
$ ansible --version
$ ansible-galaxy collection install community.general

The timezone and firewall modules used below belong to community.general, which is not part of ansible-core. Install the collection before the first playbook run.

Then make a directory for this, somewhere in your own files. It is going to become a git repository.

$ mkdir -p ~/homeserver && cd ~/homeserver

Two files to start. The first says which machines exist:

all:
  hosts:
    homeserver:
      ansible_host: 192.168.1.20
      ansible_user: admin

The second says what should be true about them:

---
- name: Baseline
  hosts: all
  become: true
  tasks:
    - name: Set the timezone
      community.general.timezone:
        name: Europe/Paris

become: true means use sudo for the tasks. hosts: all means every machine in the inventory, which for now is one.

Check that Ansible can reach the machine before asking it to change anything:

$ ansible -i inventory.yml all -m ping

That module is not an ICMP ping. It connects over SSH, runs a tiny Python program, and reports back. A SUCCESS proves your key, user and remote Python work. Test privilege escalation separately:

$ ansible -i inventory.yml all --become --ask-become-pass -m command -a whoami

The command should report root. If it does not, fix sudo before running the playbook.

Run it twice #

The first run:

$ ansible-playbook -i inventory.yml site.yml --ask-become-pass

The second, immediately after, with nothing touched in between:

$ ansible-playbook -i inventory.yml site.yml --ask-become-pass

On the first run, expect declared resources that differ to report changed. On the second, require the play recap to report changed=0. The second run still checks each task; it should find the declared state already present.

For the tasks present in this playbook, changed=0 means "I checked these declared resources, and none needed changing". It does not discover an old firewall rule whose task you removed or state you never modelled.

Move the rest of it in #

Now the real work: everything from chapters 3 and 4, as tasks.

---
- name: Baseline
  hosts: all
  become: true

  tasks:
    - name: Install automatic security updates
      ansible.builtin.apt:
        name: unattended-upgrades
        state: present
        update_cache: true

    - name: Enable the daily unattended-upgrade policy
      ansible.builtin.copy:
        dest: /etc/apt/apt.conf.d/20auto-upgrades
        owner: root
        group: root
        mode: "0644"
        content: |
          APT::Periodic::Update-Package-Lists "1";
          APT::Periodic::Unattended-Upgrade "1";

    - name: Set the hostname
      ansible.builtin.hostname:
        name: "{{ inventory_hostname }}"

    - name: Set the timezone
      community.general.timezone:
        name: "{{ server_timezone }}"

    - name: Harden the SSH daemon
      ansible.builtin.copy:
        src: files/10-hardening.conf
        dest: /etc/ssh/sshd_config.d/10-hardening.conf
        owner: root
        group: root
        mode: "0644"
        validate: /usr/sbin/sshd -t -f %s
      notify: Reload ssh

    - name: Only take security updates, and never reboot on your own
      ansible.builtin.copy:
        src: files/52-upgrades
        dest: /etc/apt/apt.conf.d/52-upgrades
        owner: root
        group: root
        mode: "0644"

    - name: Deny incoming by default
      community.general.ufw:
        direction: incoming
        policy: deny

    - name: Allow outgoing by default
      community.general.ufw:
        direction: outgoing
        policy: allow

    - name: Allow SSH from the local network
      community.general.ufw:
        rule: allow
        src: "{{ lan_subnet }}"
        port: "22"
        proto: tcp
        comment: SSH from the LAN

    - name: Turn the firewall on
      community.general.ufw:
        state: enabled

  handlers:
    - name: Reload ssh
      ansible.builtin.service:
        name: ssh
        state: reloaded

Four things in that file.

validate: on the SSH file. Before Ansible puts that file in place, it runs sshd -t against the candidate. If the file is malformed, the task fails and the old file stays. This is chapter 3's habit, automated.

notify: and handlers. The SSH task does not reload the service. It notifies a handler, which runs once at the end of the play, and only if something actually changed. Rerun with nothing to do and the service is not reloaded, because there is nothing to reload it for.

community.general prefixes. Ansible ships a small core, and everything else lives in collections. You installed this collection before the first run because even the initial timezone task needs it.

{{ lan_subnet }} is a variable, so the subnet is written once. Put it in the inventory next to the host:

all:
  hosts:
    homeserver:
      ansible_host: 192.168.1.20
      ansible_user: admin
  vars:
    lan_subnet: 192.168.1.0/24
    server_timezone: Europe/Paris

The files referenced with src: go in a files/ directory beside the playbook. Copy them from the server, or paste in what you wrote in chapter 3.

Look before you leap #

Two flags turn this from something you hope works into something you can inspect.

$ ansible-playbook -i inventory.yml site.yml --ask-become-pass --check --diff

--check asks supporting modules what they would change without applying those changes. Unsupported modules may skip work, and tasks that depend on an earlier change can report misleading results. --diff can also print file contents, including secrets, so do not capture or paste it blindly.

Not every module supports check mode faithfully, and a task that depends on an earlier task's effect can report oddly in a dry run. But for a playbook that is mostly files and firewall rules, it is close to the truth and worth running every single time.

Watch it catch drift #

Go to the server and change something by hand:

$ sudo sed -i 's/^MaxAuthTries.*/MaxAuthTries 42/' /etc/ssh/sshd_config.d/10-hardening.conf

Then, from your laptop:

$ ansible-playbook -i inventory.yml site.yml --ask-become-pass --check --diff

Require the SSH task to show a diff from the server's MaxAuthTries 42 to the repository's intended value, report a pending change, and leave the server file untouched because of --check. Drop --check only after reviewing that real diff.

Read the two header lines carefully, because they are not what you would guess. before is the file on your server. after is the file in your repository. Ansible is showing you the direction it is about to push, not two versions of the same file.

You can now ask your server whether it still matches what you meant, and correct it if not. That gets more valuable with every service you add.

Put it in git #

$ cd ~/homeserver
$ git init
$ git add inventory.yml site.yml files/
$ git commit -m "Baseline: SSH, firewall, updates"

Create the deployment directory and copy the committed configuration to the server:

$ ssh admin@192.168.1.20 'mkdir -p ~/homeserver'
$ rsync -az --exclude=.git --exclude=.env --exclude='*.key' --exclude='*.pem' \
    ~/homeserver/ admin@192.168.1.20:~/homeserver/

Run the same rsync command after later commits. It does not use --delete, so server-only certificate and rendered-secret files are not silently removed. Review stale files yourself when a configuration file is renamed.

Now you have history. Six months from now, git log tells you when you changed the firewall and, if you write decent commit messages, why.

Add a .gitignore now, while the repository is empty enough for it to be a decision rather than a cleanup:

*.key
*.pem
*.retry
.env

Note what is not excluded. Chapter 9 adds a secrets/ directory whose contents are encrypted, and commits it deliberately. What must never be committed is plaintext.

What this bought you #

A new machine is now a shorter conversation. Install the operating system, copy your key over, put the address in the inventory, run the playbook. The four things from the start of this chapter happen in about ninety seconds, in the right order, without you remembering any of them.

It is not a full rebuild. Your data is not here, and neither is anything you will install from chapter 6 onwards. But the foundation is a file now, and that file is the thing you would have been unable to reconstruct.