RAIDZ1 vs RAIDZ2 vs RAIDZ3: Choosing Parity

Once you have decided that a pool will use RAIDZ rather than mirrors, one question remains and it is close to irreversible: how many parity disks. RAIDZ1, RAIDZ2 and RAIDZ3 differ by exactly one thing in the documentation, the number of devices that can fail without data loss, but that single difference propagates into capacity, rebuild risk and how long the pool can sensibly live.

What the three levels are

The zpoolconcepts(7) man page defines them without ceremony. RAIDZ is a distributed-parity layout similar to RAID-5 and RAID-6 that does not suffer from the write hole, because copy-on-write means data and parity never become inconsistent after a power loss. raidz1 is single parity, raidz2 is double parity, raidz3 is triple parity, and the bare raidz type is an alias for raidz1.

Two sizing constraints come from the same page. A RAIDZ group with N disks of size X and P parity disks holds approximately (N − P) × X bytes and survives P devices failing. The minimum number of devices in a group is one more than the parity count, and the recommended number is between three and nine to help performance.

That recommended range is the part most people skip. A twelve-wide or sixteen-wide RAIDZ2 will build and it will work, but it sits outside what the project recommends, and the reasons show up in the two sections below.

Capacity is not simply N minus P

The (N − P) × X formula in the documentation is explicitly approximate, and the gap between it and reality widens as groups get wider.

RAIDZ does not use a consistent stripe width. Data and parity are striped across the disks in the group, and each block carries parity proportional to its own size rather than to the group’s width. Small blocks in a wide group therefore pay a much higher parity ratio than the naive formula suggests, and allocation padding adds more on top. The effect is largest exactly where people expect wide groups to shine: a pool of many small records, or zvols with a small volblocksize, spread over a wide RAIDZ vdev.

Two practical consequences follow. First, when a wide group is used for bulk sequential data, a larger recordsize on those datasets recovers much of the loss, because the parity overhead is amortised over a bigger block. Second, any capacity figure produced before the pool exists is an estimate. The zpool and RAIDZ sizer on this site applies the (N − P) formula with a fill-level margin, which is the right shape for a purchasing decision but should not be read as the number zfs list will eventually show.

Failure tolerance and the rebuild window

The headline difference is simple: RAIDZ1 tolerates one failed device per vdev, RAIDZ2 two, RAIDZ3 three. The argument for spending a disk on the next parity level is not really about the first failure, though. It is about what happens during the rebuild.

After a disk fails, the vdev is running without its full redundancy until the replacement finishes resilvering. A RAIDZ resilver has to read across the surviving members of the group to reconstruct the missing device, so its duration scales with how much data the vdev holds and how wide it is, and it competes with production traffic the entire time. On large modern disks that window is measured in days rather than hours. Everything that can go wrong during it is happening to a vdev that has already used up part of its redundancy budget.

That is why the sensible reading of the levels is by exposure rather than by capacity:

• RAIDZ1 is defensible for small groups of small disks, for data that is genuinely reproducible, or for a pool whose rebuild finishes quickly. On a wide group of high-capacity drives it leaves no margin during a multi-day resilver.
• RAIDZ2 is the reasonable default for general-purpose pools. It keeps redundancy through a full rebuild, which is what most operators actually want when they say they want redundancy.
• RAIDZ3 earns its third disk on wide groups, on very large drives, and on archival pools where a rebuild takes long enough that a second and third failure stop being hypothetical. The relative cost of the extra parity disk also falls as the group widens.

A useful reframing: choosing an extra parity disk over an extra data disk is usually cheaper than the alternative insurance, which is a faster restore path from backup.

Narrow groups beat one wide group

Random IOPS scale with the number of vdevs, not with the number of disks. A read of a record in a RAIDZ group touches every data member of that group, so a single RAIDZ vdev delivers roughly the random IOPS of one disk regardless of how many disks are in it. The dRAID documentation makes the same relationship explicit for its own layout, approximating delivered random IOPS as the number of redundancy groups multiplied by single-drive IOPS.

Twenty-four disks arranged as two twelve-wide RAIDZ2 vdevs and as four six-wide RAIDZ2 vdevs give very different pools. The second gives up capacity to parity, and returns roughly twice the random IOPS, shorter resilvers, and a smaller blast radius per vdev. It also stays inside the documented three-to-nine recommendation. For anything hosting virtual machines or databases, mirrors are usually the better answer entirely, as choosing a ZFS pool topology sets out.

dRAID, when rebuild time dominates

For large disk counts there is a third option. A dRAID vdev is built from multiple internal RAIDZ groups distributed across all children, with integrated distributed hot spares rather than idle spare drives. Because it uses a fixed stripe width, it can resilver fully sequentially, which is what makes rebuilds fast enough to matter at scale.

The tradeoffs are documented and real. The fixed stripe width raises the minimum allocation size, with the default of eight data devices and 4 KiB sectors giving a 32 KiB minimum, which can hurt both compression ratio and small-file efficiency. The documentation recommends adding a mirrored special vdev if a dRAID pool will hold many small blocks. Capacity is approximately (N − S) × (D / (D + P)) × X, where S is the number of distributed spares.

What you can change afterwards, and what you cannot

This is the part worth reading before you type zpool create.

You cannot change the parity level of an existing vdev. A RAIDZ1 vdev never becomes a RAIDZ2 vdev. The only migration is to build a new pool and send the data across.

You cannot remove a RAIDZ top-level vdev. The zpool-remove(8) man page states that top-level vdevs can only be removed if the primary pool storage contains no top-level RAIDZ or dRAID vdev, all top-level vdevs share the same ashift, and encryption keys are loaded. A RAIDZ vdev added by mistake is permanent for the life of the pool.

You can now widen a RAIDZ vdev. RAIDZ expansion, via zpool attach pool raidz2-0 <newdisk>, turns a six-wide RAIDZ2 into a seven-wide RAIDZ2 in place. The operation requires the raidz_expansion pool feature and a disk at least as large as the smallest existing member, and progress is reported by zpool status. The documented caveat matters for capacity planning: the expansion rewrites existing data across the wider group, but old blocks keep their original data-to-parity ratio, so only blocks written after the expansion use the new ratio. Expansion adds space; it does not retroactively improve the efficiency of what was already there.

You can add hot spares, and they are not a substitute for parity. A spare shortens the time before a rebuild starts. It does nothing about the exposure during the rebuild itself, which is the risk the parity level actually addresses.

Choosing in one pass

Start from the rebuild window, not the capacity table. Estimate how long a resilver will take at your disk size and expected fill level, decide how much redundancy the pool should retain for the whole of that window, and let that pick the parity level. Then check the group width against the three-to-nine recommendation, and split into more vdevs rather than widening if the count is high.

The hardware side of the same decision, including which drive properties are fixed at vdev creation, is covered in ZFS hardware requirements. What the rebuild actually looks like when it happens is in reading a DEGRADED zpool status.