ZFS Hardware Requirements: RAM, HBAs and Disks

Most filesystems are indifferent to the hardware underneath them. OpenZFS is not, because it takes on jobs that a RAID card or an array controller would normally do, and it needs unobstructed access to the disks to do them. The result is a short list of hardware decisions that genuinely matter, wrapped in a much longer list of folklore. This guide separates the two, and marks which choices you can revisit later and which ones are set the moment the pool is created.

How much RAM does ZFS actually need

ZFS uses free memory for the adaptive replacement cache, or ARC. The ARC is a cache, not an allocation you must budget for in advance, and the pool will import and function on a modest amount of memory. What more memory buys is a higher cache hit rate, which is the single most reliable way to make a pool feel fast.

By default OpenZFS does not ask you to size the ARC at all. The zfs_arc_max module parameter defaults to zero, and the documentation for zfs(4) describes what that means: the limit becomes the larger of total system memory minus 1 GiB, or five eighths of total system memory. On a 32 GB machine that works out to 31 GiB rather than the half-of-RAM figure many older guides quote. On a machine that also runs applications, virtual machines or containers, capping zfs_arc_max explicitly is usually better than letting the default compete with them.

Where the 1 GB per TB rule came from

The rule that ZFS needs 1 GB of RAM per TB of storage is repeated everywhere and is almost never qualified. It is worth noting what the OpenZFS documentation does and does not say: it states no per-TB memory requirement for ordinary use anywhere, and the only place it derives a memory figure from the size of the pool is the section on deduplication.

The OpenZFS workload tuning documentation explains the mechanism. Deduplication maintains an on-disk hash table, the DDT, with one entry per unique block in the pool, and each cached entry uses slightly more than 320 bytes of memory. The DDT has to be consulted for every dedup-able block written or freed. If it does not fit in memory, each miss becomes a random read from disk, and the documentation gives the worst case bluntly: on a single 7200 RPM drive doing roughly 100 IO/s, uncached DDT reads cap write throughput at about 100 blocks per second.

Run the arithmetic and the rule stops being mysterious. One TiB of unique data at the default 128 KiB record size is about 8.4 million blocks, so roughly 2.7 GB of DDT. The same TiB stored as 16 KiB blocks, which is typical for zvols backing virtual machines, is about 67 million blocks and roughly 21 GB. That is where “several GB of RAM per TB” comes from, and it applies only when deduplication is enabled. The documentation also describes how to stop guessing: zdb -S simulates a DDT against the data already in the pool and reports the ratio you would have achieved, and multiplying the entry count by 320 bytes gives the memory requirement directly.

With deduplication off, which is the right default for almost everyone, RAM sizing is a performance question rather than a requirement. The zpool and RAIDZ sizer on this site applies the conventional 1 GB per TB heuristic as a starting figure. Treat its RAM output as a generous upper bound for a general-purpose file server, and as a floor only if you intend to enable deduplication.

ECC memory

The OpenZFS hardware documentation states the position plainly: no technique used in ZFS, or in any other filesystem, can protect against a bit flip in memory, so ECC memory is highly recommended.

The reasoning is worth understanding rather than repeating. A flipped bit has four possible outcomes. It can land in unused memory and do nothing. It can land in something read from disk and cause a crash or a transient failure that a reboot clears. It can land in data on its way to disk, and here the timing decides everything: if the flip happens before the checksum is calculated, ZFS faithfully stores corrupt data with a valid checksum and never knows. Or it can land in on-disk metadata, where the worst case is a pool that will not import.

None of that is a ZFS-specific weakness. Every filesystem has the same exposure, and ZFS is if anything better placed because its checksums catch corruption that happens after the calculation. But ZFS is often chosen precisely for end-to-end integrity, and non-ECC memory is the gap in that chain.

Controllers: pass the disks through

The ideal ZFS controller is dull. The documentation lists three attributes: driver support on the major OpenZFS platforms, high bandwidth per port, and low cost. Support for RAID, battery backup units and hardware write caches is described as unnecessary.

Hardware RAID controllers should not be used with ZFS, and the documented reasons are concrete rather than ideological:

• Self-healing is blocked. A checksum failure on one disk is normally repaired from parity or a mirrored copy. When the controller owns the redundancy, ZFS has nothing to reconstruct from.
• Sector size information is not necessarily passed correctly on RAID 1, and cannot be passed correctly on RAID 5 or 6, which breaks the alignment work described below.
• RAID 5 and 6 reintroduce the partial-stripe write hole that ZFS avoids by design through copy-on-write.
• A failed controller may need to be replaced with the same model, or at least the same manufacturer. Disks presented natively work with any controller.
• A write cache adds a failure point that battery or flash backup only partially mitigates.
• If one array on a multi-array controller fails, the identifiers exposed to the operating system can shift, which the documentation notes has faulted pools imported from a cachefile.

Presenting each disk as a single-drive RAID 0 volume is the usual workaround and is explicitly not recommended, because most of the objections above still apply. Use a host bus adapter.

Two more interface notes from the same page. SATA disks should never be placed behind a SAS expander without an interposer, and SAS controllers should not be expected to work with SATA port multipliers. USB drives and USB-to-SATA adapters have documented problems with sector size reporting, SMART passthrough and error recovery control, and the documentation says they should be considered unreliable.

Disks: the choices that stick

Sector size and ashift. A vdev’s ashift fixes the smallest possible I/O it will issue, it is set when the vdev is created, and it cannot be changed afterwards. ZFS normally derives it from what the drive reports, which is reliable on modern drives but was not on the first generation of Advanced Format disks, which misreported 4096-byte sectors as 512 bytes for Windows XP compatibility. The practical consequence outlives the bad drives: the documentation notes that replacing a 512-byte sector drive with a 4096-byte one, in a vdev that was created with 512-byte sectors, will adversely affect performance. The reverse causes no harm. Creating pools with ashift=12 is the conventional hedge, and it is a decision you make once.

Error recovery control. This is the most underrated line item on a disk spec sheet. When a drive hits an unreadable sector it retries, and how long it is willing to retry is configurable only on drives that implement the feature: TLER on Western Digital, ERC on Seagate, CCTL on Hitachi and Samsung. Drives with it typically default to about 7 seconds. Drives without it can retry for arbitrarily long, and the documentation notes that several minutes is not impossible, during which I/O to that drive stalls. ZFS does not currently set this value itself, so on drives that support it the documentation suggests scripting a low limit at every boot.

Rotational speed and queuing. High-RPM drives buy lower seek times at a real cost in price and density, and the documentation puts the gain at no more than roughly a factor of six over slower drives. For a bulk pool, more spindles across more vdevs is usually the better spend, since random IOPS scale with vdev count rather than with the speed of any one disk.

Flash. For a separate log device, power failure protection is the property that matters, because a SLOG exists to make synchronous writes durable and a drive that loses its volatile write cache on power loss defeats the purpose.

A cache device is a different tradeoff, and the cost is quantified in the module parameters. Every buffer held on an L2ARC device needs a header in ARC, and those headers are not evicted under memory pressure, so a large cache device on a small-memory system eats the very thing it is supposed to extend. l2arc_meta_percent exists to bound this: it caps L2ARC-only headers at 33% of ARC size by default, and the documentation warns that too many headers on a system with an oversized L2ARC can render it slow or unusable. The TrueNAS guidance draws the practical line at 32 GiB, advising against adding L2ARC below it. More RAM first, then flash.

A sane order of spend

If the budget is finite, this ordering follows from the documentation rather than from taste. Redundancy inside the pool and a real backup outside it come first, because no hardware choice substitutes for either. ECC memory and enough RAM come next, because they protect and accelerate everything. A plain HBA comes third. Special, log and cache vdevs come last, and only when a specific workload justifies them.

The layout decisions those disks land in are covered in choosing a ZFS pool topology, the parity level question in RAIDZ1 vs RAIDZ2 vs RAIDZ3, and what happens when one of these disks eventually fails in reading a DEGRADED zpool status.