ZFS Pool DEGRADED: Diagnose and Replace a Failed Disk

A pool that reports DEGRADED is doing exactly what it was designed to do: it lost something and kept serving data. The dangerous part is what happens next. Most pools that are lost are not lost at the moment of the first failure, they are lost during a rushed or misdirected recovery. This is how to read the status output, work out what actually failed, replace the right device, and get back to full redundancy.

Read the whole status block, not just the state line

zpool status prints more than a health word. Every section of it carries a decision.

The pool state is one of three values in the documentation: online, degraded or faulted. An online pool has all devices operating normally. A degraded pool has had one or more devices fail but the data is still available because of redundancy. A faulted pool has corrupted metadata, or enough faulted devices that there are insufficient replicas to continue functioning.

The status and action lines are the project’s own diagnosis and recommended next step for the exact condition detected. They are worth reading before doing anything, because they distinguish cases that look identical in the tree view.

The scan line tells you whether a scrub or resilver is running, how far along it is and the estimated finish. The man page is explicit that both figures are approximate, because the amount of data and the competing workload change while the operation runs, and that on a live system scrubs and resilvers can progress beyond 100% completion, at which point no completion estimate is given.

The device tree carries per-device state plus READ, WRITE and CKSUM counters. The counters are the diagnostic information; the state word is only a summary of them.

The errors line at the bottom reports data-level damage. zpool status -v expands it into the list of files with permanent errors, which is the difference between “redundancy was consumed” and “something is actually gone”.

What each device state means

The zpoolconcepts(7) man page defines the vocabulary precisely, and the distinctions matter:

• ONLINE - functioning normally.
• DEGRADED - the device is still in use, but either checksum errors or slow I/O have exceeded acceptable levels and ZFS is flagging that something may be wrong, or I/O errors have exceeded acceptable levels but the device could not be faulted because there are not enough replicas to continue without it.
• FAULTED - the device could be opened but its contents did not match expected values, or I/O errors exceeded acceptable levels and ZFS faulted it to prevent further use.
• OFFLINE - taken offline deliberately with zpool offline.
• REMOVED - physically removed while the system was running. Detection is hardware-dependent.
• UNAVAIL - the device could not be opened at all. If the pool was imported while the device was missing, it will be identified by a unique identifier rather than by a path, because the path was never correct.

A useful implication of the DEGRADED definition: a degraded device is not necessarily a dying one. It can also mean ZFS wanted to fault the device and could not afford to.

Checksum errors are a different signal

READ and WRITE counters mean the device returned an I/O error. CKSUM counters mean something worse and more interesting: the disk returned data that was expected to be correct and was not. The documentation calls these instances of silent data corruption.

How they are attributed matters when interpreting the output. When a block is stored redundantly and a damaged copy can be reconstructed from parity or a mirror, the checksum error is reported against the disks that held the damaged data. When a block cannot be reconstructed, for instance when three disks in a RAIDZ2 group hold damage, it is not possible to determine which disks were silently corrupted, so the error is reported against every disk the block was stored on.

That is why a sudden spread of checksum errors across many devices at once should not be read as many failing disks. It usually points at something shared: a controller, an expander, a cable, a power supply, or memory. Non-ECC memory is a common culprit here, for the reasons set out in ZFS hardware requirements.

Diagnose before you replace

Replacing a healthy disk starts a resilver, and a resilver is the most stressful thing you can do to a vdev that has already lost redundancy. Spend a few minutes first.

1. Check the pattern. Errors on one device point at that device. Errors on several devices behind the same controller or expander point at the shared component.
2. Read the status and action lines. They separate “insufficient replicas” from “device has experienced errors” from “one or more devices could not be used because the label is missing or invalid”.
3. Check the drive independently. SMART attributes such as reallocated sectors, pending sectors and interface CRC errors identify a dying disk versus a bad cable. High CRC counts with no reallocations usually mean cabling, not the drive.
4. Clear and re-scrub when the cause was transient. If the trigger was a known event such as a power interruption or a cable reseat, zpool clear resets the counters, and a scrub verifies every block and repairs what redundancy allows. If the counters return, the device is the problem.

Replacing the device

The zpool-replace(8) man page describes the operation as equivalent to attaching the new device, waiting for it to resilver, and then detaching the old one. Two constraints follow from that.

The new device must be greater than or equal to the minimum size of all devices in the mirror or RAIDZ configuration. And a new-device argument is required if the pool is not redundant.

The two normal forms are:

zpool replace tank <old-device> <new-device>
zpool replace tank <device>

The second form is for the case where the failed disk has already been physically pulled and the replacement occupies the same path. ZFS recognises that this is a different disk despite the identical path, and the argument defaults to the device named.

Three flags are worth knowing. -w waits until the replacement completes rather than returning immediately. -o ashift= is the only settable property here, which is the escape hatch when a replacement drive reports a different sector size to the vdev’s original members. And -s requests sequential reconstruction, which restores redundancy as fast as possible but does not verify checksums along the way, so a scrub is started automatically when it finishes. The important limitation: sequential reconstruction is not supported for RAIDZ configurations.

Where the device path is ambiguous, zpool status -g prints vdev GUIDs instead of device names, and those GUIDs can be used directly in detach, offline, remove and replace commands. That is the safe way to name a device when several disks have shifted paths.

If a hot spare is configured, ZFS activates it automatically when a device fails. The spare is a temporary stand-in: once the original device is repaired or replaced and resilvered, the spare is released back to the pool’s spare list.

Monitor the resilver, then verify

Watch progress with zpool status, remembering that the completion estimate is approximate and can pass 100% on a busy pool. Resilvering and scrubbing are both I/O-intensive, and ZFS allows only one at a time; a scrub cannot be started while a resilver is in progress.

When the resilver finishes, the sequence that closes the incident out is:

1. Confirm the replaced device shows ONLINE and the pool state has returned to ONLINE.
2. Run zpool scrub tank. A scrub examines all data and verifies every block’s checksum, while a resilver only examines data ZFS knows to be out of date. Only the scrub tells you the rest of the vdev is intact.
3. Re-read the errors line. If permanent errors are listed, zpool status -v names the affected files and those need restoring from backup. Note that on pools with encrypted datasets, unloaded keys mean file names cannot be included in the verbose report.
4. Clear the counters with zpool clear once the cause is understood, so the next set of numbers means something.

Reducing the odds of the next one

Scrub on a schedule rather than after an incident, so that silent corruption is found while redundancy is still available to repair it. The scrub command supports pausing with -p and resuming, and -e restricts a scrub to healing only previously detected errors, which is useful for finishing an interrupted repair without a full pass.

Beyond that, the two structural levers are hardware and layout. Drives that support error recovery control avoid the multi-minute stalls that make a marginal disk look like a dead one, which is covered in ZFS hardware requirements. And the amount of redundancy left during a rebuild is fixed years earlier by the parity choice, which is the subject of RAIDZ1 vs RAIDZ2 vs RAIDZ3 and of choosing a ZFS pool topology. If a replacement is imminent and the pool may need to grow rather than swap, size the change first with the zpool and RAIDZ sizer.