Reading journalctl without drowning

The first time you run journalctl you get every log line the system has ever produced, oldest first, in a pager. It is not a helpful default and it puts a lot of people off for years.

Almost all of the value is in narrowing it down, and there are only about six ways you need to know.

Narrow by unit #

Nine times out of ten this is what you want:

$ journalctl -u ssh

-u takes a unit name. You can leave off .service. You can pass -u more than once to interleave two services, which is how you watch an application and its database at the same time.

Narrow by boot #

$ journalctl -u ssh -b

-b means "this boot only", and it is the single most useful flag on the tool. Without it you are reading history; with it you are reading the current life of the machine. -b -1 gives you the previous boot, which is how you find out what happened before a crash.

Follow it live #

$ journalctl -u ssh -f

-f follows, like tail -f. Leave it running in one terminal while you cause the problem in another. That is the fastest debugging loop available.

Stop it paging #

$ journalctl -u ssh -b --no-pager | tail -40

--no-pager prints straight to the terminal so you can pipe it. Almost every log-reading command in this book ends with | tail -40, because the end is where the recent problem lives.

Narrow by time #

$ journalctl --since "10 min ago"
$ journalctl --since "yesterday" --until "today"

--since and --until understand both timestamps and plain English like yesterday, today, 1 hour ago. When you know roughly when something broke but not which service caused it, this is how you find out.

Narrow by severity #

$ journalctl -p err -b

-p filters by priority, and err means errors and worse. This is a good five-second health check on a machine you have not looked at in a while. Be aware that plenty of software logs alarming things at warning and genuinely broken things at info, so this narrows rather than decides.

Putting them together #

The combination worth memorising, because it answers "what is wrong with this service right now":

$ systemctl status ssh
$ journalctl -u ssh -b --no-pager | tail -40

systemctl status gives you the current state, whether it is enabled at boot, its main process, and the last few log lines. Then the journal gives you the rest of the story. In that order.

Reading what you find #

A few things that confuse people early on.

Messages can be rate-limited. A suppression notice means some repeated events were dropped. Inspect the service and journald rate-limit settings before treating the visible count as the event count.

The first error is the one that matters. Software tends to fail, then fail again in more visible ways because of the first failure. Scroll up to where it started going wrong rather than reading the loudest line.

Timestamps use the selected output timezone. Use journalctl --utc when correlating systems in different zones, and record the offset with any incident notes.

Not everything is in the journal. Some software writes under /var/log; Docker application output is normally read with docker compose logs and, after chapter 14, copied into Loki. Confirm the configured log driver before deciding that silence means no event occurred.

Keeping it from eating the disk #

The journal is capped, but the cap may be generous. Check what it is using:

$ journalctl --disk-usage

If it is larger than you would like, SystemMaxUse in /etc/systemd/journald.conf sets a limit, and journalctl --vacuum-time=14d clears out anything older than a fortnight. This matters more than it sounds on a machine whose disk you also want to fill with photographs.