Choosing a ZFS Pool Topology: Mirrors, RAIDZ and vdev Layout

ZFS gives you a small number of layout decisions that are effectively permanent, surrounded by a large number of tunables you can change at any time. Knowing which is which is most of what pool design requires.

A pool is a stripe over vdevs

The pool is not a redundancy group. Its virtual devices are. A pool distributes writes across all of its vdevs, and it remains available only while every vdev remains available. If one vdev fails beyond its own redundancy, the pool is gone, including data that physically resides on other vdevs. Every layout question follows from that.

Mirrors

A mirror vdev keeps identical copies on each of its members. Read requests can be served by any member, so random read performance scales with the number of vdevs. Rebuilding a replaced disk means copying only the data that exists, from a single surviving member, which is the fastest recovery path available. The cost is usable capacity, since a two way mirror gives you half.

Mirrors are the usual answer for random workloads such as virtual machine disks and databases, and for pools you expect to expand incrementally, because adding one more small mirror vdev is cheap.

Mirror against RAIDZ at a glance

	Mirror vdev	RAIDZ vdev
Usable capacity	1/n of raw for an n-way mirror	Approximately (N − P) × disk size
Failure tolerance	n − 1 members per vdev	P disks per vdev, P = 1, 2 or 3
Random read IOPS	Scales with member count and vdev count	Roughly one disk per vdev
Rebuild method	Copy allocated data from a surviving member	Reconstruct from every surviving member
Rebuild stress	Lowest available	Highest, and scales with width and fill
Widening in place	Attach another member to the mirror	RAIDZ expansion, one disk at a time
Removing the vdev later	Supported for mirrored top-level vdevs	Not supported
Best fit	Virtual machines, databases, incremental growth	Bulk and sequential storage, capacity per parity disk

Estimate the capacity side of that table for a specific disk count with the zpool and RAIDZ sizer, then read the parity column against RAIDZ1 vs RAIDZ2 vs RAIDZ3, which covers what each level costs and which parts of the decision are reversible.

RAIDZ

A RAIDZ vdev spreads data and parity across its members, with one, two or three parity disks depending on the level. It delivers far better usable capacity per parity disk than mirroring, and it is well suited to sequential workloads and bulk storage.

The tradeoff is that a RAIDZ vdev provides roughly the random IOPS of a single disk, because a read of a record touches every member. Wide RAIDZ groups also concentrate a lot of data behind one redundancy group. Space efficiency is not the simple formula many people assume either: parity and padding overhead depend on record size relative to the group width, so small records in a wide group waste more than the naive calculation suggests.

Resilver time is the real risk factor

The danger period after a failure is however long the rebuild takes, and reading a DEGRADED zpool status walks through what that period looks like in practice. A mirror resilver reads one surviving copy. A RAIDZ resilver has to read across the entire group to reconstruct, so it scales with how full and how wide the vdev is, and it competes with production traffic the whole time. Wider groups and larger disks lengthen the window in which a second failure is fatal. Choosing an extra parity disk over an extra data disk is often the cheaper insurance.

Support vdevs are not optional extras

A separate log device only accelerates synchronous writes. Most file sharing workloads do not issue them, so it does nothing for typical use. A cache device consumes memory to index its own contents and only helps when the working set is much larger than what memory can hold, so adding one on a memory constrained system can make things worse. A special vdev holds metadata and, if configured, small blocks. It is part of the pool, not a cache, and losing it destroys the pool, so it must carry redundancy matching the rest of the design.

More memory improves read performance more reliably than any of these devices, because the adaptive replacement cache uses whatever is free. How much memory that means in practice, and where the 1 GB per TB rule actually comes from, is covered in ZFS hardware requirements.

Settings that are fixed at creation

Sector size alignment is set per vdev when it is created and cannot be changed afterward. Getting it wrong on disks that report a smaller logical sector than they physically use causes permanent write amplification. Record size and compression, by contrast, are dataset properties that apply to newly written data, so they can be adjusted later, but existing blocks keep whatever they were written with.

Keep scrubs running

A scrub reads and verifies checksums across the pool, repairing damage while redundancy is still available. Silent corruption discovered during a resilver is discovered too late.