When a disk starts to fail

Some failures produce warning counters; others are sudden. SMART is an early signal, not a guarantee.

Reading the drive #

smartctl and its scheduling daemon smartd come from the smartmontools package. nvme smart-log comes from nvme-cli:

$ sudo apt install smartmontools nvme-cli

$ sudo smartctl -a /dev/sda
$ sudo smartctl -H /dev/sda

-H gives the drive's own overall verdict, which is worth knowing and not worth trusting: drives pass their self-assessment right up until they do not.

For NVMe:

$ sudo smartctl -a /dev/nvme0
$ sudo nvme smart-log /dev/nvme0

The attributes that matter #

SMART names and raw values are vendor-specific. Read the drive manufacturer's documentation where available, and consider both a bad absolute value and change over time.

For spinning disks:

• Reallocated_Sector_Ct. Sectors replaced from a spare pool. A stable non-zero raw value is evidence to investigate; a growing value is a strong replacement signal.
• Current_Pending_Sector. Sectors the drive cannot read and has not yet reallocated. These are unreadable data right now. Worse than reallocated.
• Offline_Uncorrectable. Could not be read or corrected. Treat like pending.
• Reported_Uncorrect. Errors the drive could not fix.

For SSDs and NVMe:

• Percentage_Used or Wear_Leveling_Count. How much of the rated write endurance is consumed. Predictable and slow; plan around it rather than reacting.
• Media_and_Data_Integrity_Errors. Should be zero. Any value is a reason to look closer.
• Available_Spare. Falling towards its threshold means the drive is running out of replacement blocks.

Temperature matters less than people expect within normal ranges, and a lot at extremes.

Test it rather than waiting #

$ sudo smartctl -t short /dev/sda     # a couple of minutes
$ sudo smartctl -t long /dev/sda      # hours; reads the whole surface
$ sudo smartctl -l selftest /dev/sda  # the results

A long test reads every sector, which is how pending sectors get discovered rather than waiting for you to read that file.

Choose a schedule appropriate to the drive, workload and vendor guidance. Do not start a long test while recovery copying or heavy application work is running. smartd can schedule tests and notifications.

Which numbers to alert on #

Chapter 14's monitoring can watch these, but node_exporter has no SMART collector of its own. Two options: run smartctl_exporter alongside it, or use the community smartmon.sh script on a timer, writing into node_exporter's --collector.textfile.directory.

Alert on change, not on level:

• Reallocated sectors increased since last check.
• Pending sectors above zero.
• Available spare below the drive's own threshold.
• SSD percentage used crossing 80.

A stable historical reallocation and a rapidly increasing count are different signals. Treat a change alongside health, error-log and backup evidence rather than relying on one universal number.

What to do when one moves #

Check your backups first. Before touching anything, confirm the last backup succeeded and run the restore drill. A failing disk is exactly when you find out whether your backups worked, and doing that discovery before you stress the drive is the whole point.

Do not run a long test on a drive that is actively failing if you have not got a good copy. Reading the whole surface can be the thing that finishes it.

Replace rather than nurse. A drive costs less than the afternoon. If pending sectors are non-zero and rising, it is done.

Copy with a tool that keeps going. GNU ddrescue records progress in a map file, skips unreadable regions and can retry them later. Read its manual and recover to a different healthy device; source and destination order is a destructive distinction.

The quiet failure mode #

The dangerous case is not a disk that dies. It is a disk that returns wrong data occasionally without reporting an error.

Filesystem checksums detect this: ZFS and Btrfs verify checksummed blocks on read. Repair still requires a valid redundant copy or backup. ext4 does not checksum ordinary file data. Chapter 7 turns on PostgreSQL data checksums so the database can detect some damaged pages even though the underlying ext4 file data has no equivalent check.

If silent corruption is part of your threat model, choose a checksumming filesystem together with scrub scheduling, redundancy and tested backups. Detection alone cannot reconstruct a good block.