Choosing a ZFS Pool Topology: Mirrors, RAIDZ and vdev Layout
How vdev choice fixes a ZFS pool's redundancy, IOPS and resilver window, plus which settings are permanent once the pool is created.
ZFS gives you a small number of layout decisions that are effectively permanent, surrounded by a large number of tunables you can change at any time. Knowing which is which is most of what pool design requires.
A pool is a stripe over vdevs
The pool is not a redundancy group. Its virtual devices are. A pool distributes writes across all of its vdevs, and it remains available only while every vdev remains available. If one vdev fails beyond its own redundancy, the pool is gone, including data that physically resides on other vdevs. Every layout question follows from that.
Mirrors
A mirror vdev keeps identical copies on each of its members. Read requests can be served by any member, so random read performance scales with the number of vdevs. Rebuilding a replaced disk means copying only the data that exists, from a single surviving member, which is the fastest recovery path available. The cost is usable capacity, since a two way mirror gives you half.
Mirrors are the usual answer for random workloads such as virtual machine disks and databases, and for pools you expect to expand incrementally, because adding one more small mirror vdev is cheap.
RAIDZ
A RAIDZ vdev spreads data and parity across its members, with one, two or three parity disks depending on the level. It delivers far better usable capacity per parity disk than mirroring, and it is well suited to sequential workloads and bulk storage.
The tradeoff is that a RAIDZ vdev provides roughly the random IOPS of a single disk, because a read of a record touches every member. Wide RAIDZ groups also concentrate a lot of data behind one redundancy group. Space efficiency is not the simple formula many people assume either: parity and padding overhead depend on record size relative to the group width, so small records in a wide group waste more than the naive calculation suggests.
Resilver time is the real risk factor
The danger period after a failure is however long the rebuild takes. A mirror resilver reads one surviving copy. A RAIDZ resilver has to read across the entire group to reconstruct, so it scales with how full and how wide the vdev is, and it competes with production traffic the whole time. Wider groups and larger disks lengthen the window in which a second failure is fatal. Choosing an extra parity disk over an extra data disk is often the cheaper insurance.
Support vdevs are not optional extras
A separate log device only accelerates synchronous writes. Most file sharing workloads do not issue them, so it does nothing for typical use. A cache device consumes memory to index its own contents and only helps when the working set is much larger than what memory can hold, so adding one on a memory constrained system can make things worse. A special vdev holds metadata and, if configured, small blocks. It is part of the pool, not a cache, and losing it destroys the pool, so it must carry redundancy matching the rest of the design.
More memory improves read performance more reliably than any of these devices, because the adaptive replacement cache uses whatever is free.
Settings that are fixed at creation
Sector size alignment is set per vdev when it is created and cannot be changed afterward. Getting it wrong on disks that report a smaller logical sector than they physically use causes permanent write amplification. Record size and compression, by contrast, are dataset properties that apply to newly written data, so they can be adjusted later, but existing blocks keep whatever they were written with.
Keep scrubs running
A scrub reads and verifies checksums across the pool, repairing damage while redundancy is still available. Silent corruption discovered during a resilver is discovered too late.