A safe front door

This chapter changes your machine, starting with the one thing that can lock you out of it.

What a key actually does #

Keys are better than passwords, and the reason explains several later chapters.

A password is a shared secret. To prove you know it, you send it. That means the secret reaches the far end on every single login, and if the far end is not what you thought it was, it now has something it can reuse for as long as the password lives.

A key is a pair of numbers. The private half stays on your laptop in a file, and the public half goes on the server in a list of keys allowed to log in as you. When you connect, the server sends a random challenge, your client signs it with the private key, and the server checks that signature against the public half. The private key never crosses the wire, and the signature is worthless afterwards because the next challenge will be different.

Two consequences. Nobody can guess a key, so the automated password guessing that every internet-facing SSH port receives stops mattering. And your private key file is now the thing that matters, which means it deserves a passphrase and a backup.

Make a key on your laptop #

On your own machine, not the server:

$ ssh-keygen -t ed25519 -C "you@your-laptop"

The command prompts for the file path and passphrase. The comment after -C is a label stored in the public key. Make it say which machine the key lives on, because later it will distinguish this entry from other authorised keys.

Accept the default path. About the passphrase: use one. It encrypts the private key file at rest, so a stolen laptop is not automatically a stolen server. The usual objection is that you would then type it constantly, and the answer to that is the ssh agent (man page), which holds the decrypted key in memory for the rest of your session:

$ ssh-add ~/.ssh/id_ed25519

On macOS and most Linux desktops an agent is already running and your first connection will offer to remember the passphrase. You type it once per login to your laptop, not once per connection.

You now have two files. id_ed25519 is the private key, and it should be readable only by you. id_ed25519.pub is the public half, and it is not secret at all.

Give the server your public key #

The polite way, from your laptop:

$ ssh-copy-id admin@192.168.1.20

That logs in using whatever method currently works, appends your public key to ~/.ssh/authorized_keys on the server, and fixes the permissions. If your installer already put a key on the machine for you, this may be a no-op, and you can check with ssh-copy-id -n first to see what it would do.

If password login is already disabled and you have no key on the machine yet, ssh-copy-id has no way in. You will need console access, or your cloud provider's key-injection mechanism. That is the chicken-and-egg problem the lockout annex covers.

Doing it by hand is worth seeing once, because the permissions are the part that goes wrong. On the server:

$ mkdir -p ~/.ssh && chmod 700 ~/.ssh
$ nano ~/.ssh/authorized_keys        # paste the .pub line, one key per line
$ chmod 600 ~/.ssh/authorized_keys

Prove it works before you change anything #

Open a new terminal and connect. Verbosely, because you want to see which method it used:

$ ssh -v admin@192.168.1.20 2>&1 | grep -i "offering\|authentication succeeded"

Authentication succeeded (publickey) is the line you are looking for. If it says (password) instead, your key is not being accepted and you must fix that before the next section, because the next section removes passwords.

Read what the server is actually running #

Most SSH hardening guides tell you to edit a file, and not that the file might be ignored.

Look at the configuration as sshd sees it, not as you wrote it:

$ sudo sshd -T | grep -Ei 'passwordauth|permitroot|pubkeyauth|kbdinteractive'

sshd -T prints the effective configuration: every setting, resolved, after all includes. Every one of them is documented in the sshd_config manual. Run it before and after every change in this section.

Now look at why the answer can be surprising:

$ grep -n -i '^include' /etc/ssh/sshd_config
$ sudo ls -l /etc/ssh/sshd_config.d/

Two rules govern how those combine, and together they surprise nearly everyone:

1. For most keywords, the first value sshd finds wins. Not the last. This is the opposite of how nearly every other config format on your machine behaves.
2. Drop-in files are read in filename order, and the Include line sits at a particular place in the main file, so anything written above line 24 also gets first refusal.

So a late-sorting hardening file can be silently overruled by an earlier drop-in that already set the same keyword. A file can be present and syntactically correct while having no effect, which is why the effective sshd -T output is the control.

The fix is to make your file sort first. Look at the names already present and pick a lower number.

Write your own drop-in #

Create the file on the server:

$ sudo nano /etc/ssh/sshd_config.d/10-hardening.conf

# /etc/ssh/sshd_config.d/10-hardening.conf
#
# The number matters. sshd uses the FIRST value it finds for most keywords, and
# drop-ins are read in filename order, so a file that sorts earlier wins.
# Check the result with:  sudo sshd -T | grep -Ei 'passwordauth|permitroot'

# Only keys. No passwords, no keyboard-interactive fallback.
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Nobody logs in as root, ever. Use your own account and sudo.
PermitRootLogin no

# Fewer guesses per connection, and fewer idle half-open connections.
MaxAuthTries 3
LoginGraceTime 20

# Turn off things a headless server has no use for. Each one is a feature that
# cannot be abused if it is not enabled.
X11Forwarding no
AllowAgentForwarding no
PermitTunnel no

# "local" still lets you run `ssh -L 8080:localhost:8080` from your laptop to
# peek at something on the server, which is genuinely useful later. It refuses
# the reverse direction, which would let a session open a door back in.
AllowTcpForwarding local

# Only these accounts may log in at all.
AllowUsers admin

Be careful with the AllowUsers line: it is a whitelist, and a typo in it locks out everybody. Check that the name shown is genuinely yours before you save the file, and if you would rather not take the risk at all, leave the line out. The other settings carry most of the value.

Then, in this order, every time:

$ sudo sshd -t
$ sudo systemctl reload ssh
$ sudo sshd -T | grep -Ei 'passwordauth|permitroot'

Require passwordauthentication no and permitrootlogin no in your real output before testing a new login.

sshd -t parses the configuration and says nothing at all if it is valid. Running it before a reload catches a typo before it takes your SSH service down.

Now the actual test, from your laptop, in a new terminal:

$ ssh admin@192.168.1.20

If that works, you may close the safety session. Not before.

Attack it on purpose #

Try to do the thing you just forbade:

$ ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password admin@192.168.1.20

Require the attempt to be refused without a password prompt. If the final diagnostic lists publickey as the only accepted method, the server had no password method to try.

Then watch it from the server's side. In your other session:

$ sudo journalctl -u ssh -b --no-pager | tail -20

You will see your rejected attempt. Learn this view while the source and timestamp are known. This book never publishes SSH to the internet, so later unexpected failures are evidence to investigate rather than assumed background noise.

While you are in here #

Three small things that belong to the front door, and take two minutes each.

Root login. Your drop-in already set PermitRootLogin no. Note what the default often is: prohibit-password, which permits root to log in with a key. That is not unreasonable, but a named human account plus sudo gives you an audit trail that root cannot.

sudo itself. It lets a permitted user run a command as another user, normally root, and it logs each use. When a command fails with a permission error, the instinct to prefix it with sudo is usually right and occasionally a mistake worth noticing: a file you cannot read as yourself may be one you should not be editing.

Name and time. From the last chapter's survey you may have found the machine on Etc/UTC with a hostname you did not choose.

$ sudo hostnamectl set-hostname homeserver
$ sudo timedatectl set-timezone Europe/Paris
$ timedatectl

UTC is a perfectly good choice for a server and arguably the better one for reading logs. Local time is friendlier when the logs are about your own family's photo uploads. Pick one deliberately and write down which, because later services will inherit it and inconsistency between them is genuinely confusing.

Updates that install themselves #

The most likely way this machine gets compromised is a known vulnerability in a package that had a patch four months ago.

Debian and Ubuntu provide unattended-upgrades for exactly this. Install and enable it before inspecting the policy:

$ sudo apt update
$ sudo apt install unattended-upgrades
$ sudo dpkg-reconfigure -plow unattended-upgrades

The last command creates the periodic configuration when the installer did not enable it. Now check the result:

$ systemctl list-timers --all | grep -i apt
$ cat /etc/apt/apt.conf.d/20auto-upgrades

Two timers do the work: one refreshes the package lists, the other applies upgrades. Require the file to set both Update-Package-Lists and Unattended-Upgrade to "1", meaning daily. If either is absent, do not continue as though updates are enabled; rerun dpkg-reconfigure and inspect its output.

That tells you it runs. It does not tell you what it will install, which is the part worth being deliberate about:

$ grep -v '^\s*//' /etc/apt/apt.conf.d/50unattended-upgrades | grep -v '^$' | head -30

What you want is the distribution's maintained security origins, including any Ubuntu Pro security pockets the machine is entitled to, without enabling general feature updates. Do not replace that list with a single hand-written origin: doing so can silently exclude security coverage as repository names evolve.

Read the uncommented Allowed-Origins and Origins-Pattern entries in the packaged file and compare them with apt-cache policy. Leave the distribution's security list intact. Add only the local reboot choice:

$ sudo nano /etc/apt/apt.conf.d/52-upgrades

// Never reboot on its own. A server that reboots at 03:00 in the middle of a
// photo import, without you knowing why it went away, costs more than the
// pending kernel patch.
Unattended-Upgrade::Automatic-Reboot "false";

Now inspect the merged configuration rather than assuming which earlier files won:

$ apt-config dump Unattended-Upgrade::Origins-Pattern
$ apt-config dump Unattended-Upgrade::Allowed-Origins

Confirm that every enabled origin is a security source you recognise, including entitled ESM security sources where applicable, and that ordinary update pockets are not enabled. Then run a dry run; it does not install packages:

$ sudo unattended-upgrade --dry-run --debug 2>&1 | tail -20

Done #