ZFS Cheat Sheet and Guide

Last updated: December 19th, 2023 - Referencing OpenZFS v2.2.2

ZFS is a magical filesystem created by Sun Microsystems, with an initial release of 2006. While ZFS may be almost 20 years old by this post (over 22 if you count when they started development), it was ahead of it's time then, and even now, with newer filesystems like BTRFS, APFS, and ReFS, it is still ahead of its time. In just about every aspect, ZFS has more features, better data integrity, and the most amount of data and files possible on a single filesystem.

If you are storing data on servers, especially across drives in an array, you should be considering ZFS, unless it is not possible on the hardware or OS you are running.

Below I will group the Getting Started information in expandable info boxes.

While this guide is detailed enough for almost all cases, if you need even more information on doing more advanced administration, please refer to the OpenZFS Documentation, The FreeBSD ZFS Documentation, or even the Oracle ZFS Documentation

The examples below will assume you are doing this on Linux but ZFS commands themselves are universal. ZFS requires root privileges so either be root or use sudo/doas/etc

Creating a ZFS Pool

Identifying Drives

When using drives for ZFS, you need to utilize their actual drive names or WWNs. We also need to check for the sector sizes these drives support.
While you can create Pools with drive partitions, it is best to use the whole drive when possible.

To start, we'll take an inventory of the drives available.

lsblk -o NAME,SIZE,MODEL,FSTYPE,LOG-SEC,PHY-SEC,SERIAL,MOUNTPOINT

And we get an output similar to this, but I am including only the drives I want to use in this example.

NAME         SIZE  MODEL               FSTYPE         LOG-SEC PHY-SEC
sdc          14.6T ST16000NM000J-2TW103                    512    4096
sdf          14.6T ST16000NM000J-2TW103                    512    4096
sdg          14.6T ST16000NM000J-2TW103                    512    4096
sdh          14.6T ST16000NM000J-2TW103                    512    4096
sdi          14.6T ST16000NM000J-2TW103                    512    4096

From this information, we know we want to get the drive ID names of sdc, sdf, sdg, sdh, and sdi. As because these are 4K native drives, we can include ashift=12 in our pool creation command to force 4K sector mode. If it just says 512 or otherwise, leave out the ashift in the creation command. But ZFS is smart and it will check the reported Physical Block Size of the drive and choose what's best, so this is almost never needed.

So we will check the /dev/disk/by-id/ directory in Linux to see what the drive IDs are. We can use grep to filter the output easier.

ls -lh /dev/disk/by-id | grep -e sdc -e sdf -e sdg -e sdh -e sdi       
lrwxrwxrwx 1 root root  9 Jan  9 00:38 ata-ST16000NM000J-2TW103_ZR52CQQK -> ../../sdi
lrwxrwxrwx 1 root root  9 Jan  9 00:38 ata-ST16000NM000J-2TW103_ZR57LBNF -> ../../sdh
lrwxrwxrwx 1 root root  9 Jan  9 00:38 ata-ST16000NM000J-2TW103_ZR5CZTZ6 -> ../../sdg
lrwxrwxrwx 1 root root  9 Jan  9 00:38 ata-ST16000NM000J-2TW103_ZR5D4D23 -> ../../sdf
lrwxrwxrwx 1 root root  9 Jan  9 00:38 ata-ST16000NM000J-2TW103_ZR60QV5P -> ../../sdc
lrwxrwxrwx 1 root root  9 Jan  9 00:38 wwn-0x5000c5007443c744 -> ../../sdf
lrwxrwxrwx 1 root root  9 Jan  9 00:38 wwn-0x5000c500e3c9cfab -> ../../sdh
lrwxrwxrwx 1 root root  9 Jan  9 00:38 wwn-0x5000c500e53da9b9 -> ../../sdc
lrwxrwxrwx 1 root root  9 Jan  9 00:38 wwn-0x5000c500e5c8b8a5 -> ../../sdg
lrwxrwxrwx 1 root root  9 Jan  9 00:38 wwn-0x5000c500e5e2bfee -> ../../sdi

We want ideally the ones that look like ata-DRIVEMODEL_SERIAL in this case, but if you are using NVMe drives, they will start with nvme-DRIVEMODEL_SERIAL. wwn- entries will also be acceptable but not all drives use those and this creates extra steps in identifying bad drives later. So for now, copy and paste those names somewhere.

For the following, replace POOLNAME with your desired name of the Pool storage. It is case sensitive and can not be the following: mirror, raidz, spare, draid, log

Replace drive#-name-serial with the actual Drive ID names you found earlier.

DO NOT USE THE TYPICAL NAMES OF DRIVES IN THE FORMAT OF sda or /dev/sda or nvme0n1 or /dev/nvme0n1or one day you will find after a reboot or a change of cables, your drives will have different names and the ZFS pool will go offline. This can be fixed if you do this, later in this guide.

Single Drive Pool

• Min/Max Drives: 1/1
• Fault Tolerance: None
• Drive Space Overhead: None
• Read / Write Speed: Native Drive Speed

zpool create POOLNAME drive1-name-serial

Striped (RAID 0) Pool

• Min Drives: 1
• Fault Tolerance: None
• Drive Space Overhead: None
• Read / Write Speed: Native speed multiplied by # of Drives

zpool create POOLNAME drive1-name-serial drive2-name-serial

Mirrored (RAID 1) Pool

• Min Drives: 2
• Fault Tolerance: (# of Drives) - 1
• Drive Space Overhead: ((# of Drives) - 1) / (# of Drives)
• Read / Write Speed: 2x Drives / Native Drive Speed

zpool create POOLNAME mirror drive1-name-serial drive2-name-serial

Example of a real command

zpool create VENTURE mirror nvme-T-FORCE_TM8FP8002T_112104060080114 nvme-T-FORCE_TM8FP8002T_112104060080143

Striped + Mirrored (RAID 10) Pool

• Min Drives: 4
• Fault Tolerance: (# of Drives in each mirror) - 1
• Drive Space Overhead: ((# of Drives) x Parity Stripes over # of Mirrors
• Read / Write Speed: Typically 2x Drives / 2x Drives - Read speed increases with # of mirror groups

zpool create POOLNAME mirror drive1-name-serial drive2-name-serial mirror drive3-name-serial drive4-name-serial

Example of a real command

zpool create ENTERPRISE mirror ata-ST16000NM000J-2TW103_ZR52CQQK ata-ST16000NM000J-2TW103_ZR57LBNF mirror ata-ST16000NM000J-2TW103_ZR5CZTZ6 ata-ST16000NM000J-2TW103_ZR5D4592 mirror ata-ST16000NM000J-2TW103_ZR5D4D23 ata-ST16000NM000J-2TW103_ZR60QV5P

You may have noticed how the syntax here shows that a RAID 10 is made up of 2 mirror commands, with a striped layout of the mirrors. This also matters how you add drives to these Pools later.

Here's an example of what a RAID 10 ZFS Pool looks like after created.

zpool status
  pool: CHALLENGER
 state: ONLINE
  scan: scrub repaired 0B in 00:10:24 with 0 errors on Sun Jan  8 00:34:25 2023
config:

	NAME                                      STATE     READ WRITE CKSUM
	CHALLENGER                                ONLINE       0     0     0
	  mirror-0                                ONLINE       0     0     0
	    nvme-TEAM_TM8FP4004T_112211290580504  ONLINE       0     0     0
	    nvme-TEAM_TM8FP4004T_112211290580505  ONLINE       0     0     0
	  mirror-1                                ONLINE       0     0     0
	    nvme-TEAM_TM8FP4004T_112211290580958  ONLINE       0     0     0
	    nvme-TEAM_TM8FP4004T_112211290581167  ONLINE       0     0     0

errors: No known data errors

RAID-Z Type Pools

These are more specialized for ZFS. These will provide the most usable storage space, the best redundancy, the best data integrity, but also the slowest write speeds for ZFS Pool arrays. To calculate storage and speed, use this ZFS RAID-Z Calculator.

Which RAID-Z level should I choose?

You will have to decide based on your needs and your drives. The safest best is to choose RAID-Z2 by default. If you value your data, then RAID-Z3. RAID-Z1 can be useful if you're on a budget and your ability to quickly change out a failed drive. SSDs can work well on RAID-Z1 due to how they most likely won't die from resilvering a new drive replacement and can do it very quickly, as well as maximizing the space utilization.

If you're running high IO intensive applications like databases on ZFS, you may want to consider ZFS RAID 10 and not a RAID-Z, since it will have the best write speeds.

RAID-Z Pool (RAID-Z1)

• Min Drives: 3
• Fault Tolerance: 1 Drive
• Drive Space Overhead: 1 Drive
• Read / Write Speed: # of Drives - 1 / Single Drive Speed

zpool create POOLNAME raidz drive1-name-serial drive2-name-serial drive3-name-serial drive4-name-serial ...

Example of a real command

zpool create INTREPID raidz1 ata-TEAM_T2532TB_112208190540104 ata-TEAM_T2532TB_112208190540533 ata-TEAM_T2532TB_112208290490136 ata-TEAM_T2532TB_112208190540057

RAID-Z2 Pool

• Min Drives: 4
• Fault Tolerance: 2 Drives
• Drive Space Overhead: 2 Drives
• Read / Write Speed: # of Drives - 2 / Single Drive Speed

zpool create POOLNAME raidz2 drive1-name-serial drive2-name-serial drive3-name-serial drive4-name-serial ...

RAID-Z3 Pool

• Min Drives: 5
• Fault Tolerance: 3 Drives
• Drive Space Overhead: 3 Drives
• Read / Write Speed: # of Drives - 3 / Single Drive Speed

zpool create POOLNAME raidz3 drive1-name-serial drive2-name-serial drive3-name-serial drive4-name-serial drive5-name-serial ...

Example of a RAID-Z type Pool in Status output

  pool: INTREPID
 state: ONLINE
  scan: scrub repaired 0B in 00:52:52 with 0 errors on Sun Jan  8 01:16:54 2023
config:

	NAME                                  STATE     READ WRITE CKSUM
	INTREPID                              ONLINE       0     0     0
	  raidz1-0                            ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208190540104  ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208190540533  ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208290490136  ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208190540057  ONLINE       0     0     0

errors: No known data errors

After Creating Your Pool

Now that your Pool is created, you need to set your default global properties for it and then a mountpoint. Check below for how to do that in the property flags section.

I personally like to set these properties on my Pools at the Pool level before I do anything with the Pool.

# Set ZStandard Compression globally for space and sped up reads
zfs set compression=zstd-7 POOLNAME

# Disable atime (Access Time) for performance and reduce SSD wear
zfs set atime=off POOLNAME

# Set BLAKE3 Checksums for performance when writing
zfs set checksum=blake3 POOLNAME

# Set the mountpoint
zfs set mountpoint=/storage/POOLNAME POOLNAME

# Create Datasets for major directories/categories
zfs create POOLNAME/DATASET_NAME

# Recordsize Adjustment OPTIONAL FOR MEDIA FILES
zfs set recordsize=1M POOLNAME/DATASET_NAME

If you looked at the ZFS Terminology area, you'll see that Datasets are virtual filesystems in ZFS. They show up as directories in the Pool. There's a lot of benefits to utilizing Datasets for your data.

Datasets have their own properties

Datasets inherit properties set from their parent Pool or parent Dataset. But you can set properties specifically on Datasets so that they have addition or changed features. Such as but not limited to:

• Different level of compression - Maybe you want more or less compression for that Dataset for the type of data it is holding
• Quotas - You can set limits of how much data that Dataset can have. This is useful for many situations including setting a user's home directory as a Dataset to limit their disk space utilization.
• ATIME - Maybe the data in that Dataset benefits from Access Times
• Encryption - You can specifically encrypt the data in a Dataset
• SAMBA Mode - You can set a property that makes the Dataset SAMBA friendly for access from Windows machines that are case insensitive
• Checksum type - You can change what checksum algorithm is used for that data
• Mountpoint - You can set a mountpoint for that Dataset to be somewhere else in the Filesystem

Datasets show disk space utilization info instantly

With the zfs list command, you can see live information instantly about how much space is being utilized by the Datasets. No more running and waiting for du -h on multiple directories to see what's using up the disk space.

Datasets can be Snapshot and Recovered

You can snapshot a Dataset instantly and rollback to that snapshot anytime. Snapshots only use space for file differences or deltas between the snapshot and the live Dataset filesystem. This is great for backing up the existing data before you make major changes to the data. Or you can have a script that snapshots the Dataset regularly incase one day you accidently delete or change a file. You can then mount that snapshot to recover the data, roll back the snapshot (overwriting the new data), or even access the snapshot files from the hidden folder in the Dataset .zfs/snapshots/snapshot_name

Datasets can be Sent to other ZFS Pools locally and remotely

In conjunction with the Snapshot feature, you can send the Snapshot of a Dataset to another pool and retain all the properties and permissions, and it only sends the difference in files, sort of like a ZFS specific RSYNC.

Create a ZFS Dataset

Reminder: Datasets are case sensitive, name them how you'd name a directory

# Dataset at the root of the Pool
zfs create POOLNAME/DATASET_NAME

# Dataset in another Dataset
zfs create POOLNAME/DATASET_NAME/ANOTHER_DATASET

Rename a ZFS Dataset

zfs rename POOLNAME/CURRENT_DATASET_NAME POOLNAME/NEW_DATASET_NAME

Destroy a ZFS Dataset

WARNING: THIS WILL DELETE ALL THE DATA IN THE DATASET INSTANTLY.

If there are children Datasets under this Dataset, you will need to specify -r

# Destroy a Dataset (no children Datasets)
zfs destroy POOLNAME/DATASET_NAME

# Destroy Dataset and all children Datasets underneath
zfs destroy -r POOLNAME/DATASET_NAME

Destroy a ZFS Dataset Snapshot

Same principle as the Dataset, but you specify the Snapshot only

# Destroy a Dataset Snapshot
zfs destroy POOLNAME/DATASET_NAME@SNAPSHOT_NAME

# Recursively destroy all Snapshots of Dataset and Children Dataset Snapshots with the same name
zfs destroy -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME

ZFS properties

See all properties for a POOL or DATASET

zfs get all POOLNAME
zfs get all POOLNAME/DATASET_NAME

See specific properties for a POOL or DATASET

# Single Property
zfs get propertyA POOLNAME
zfs get propertyA POOLNAME/DATASET_NAME

# Multiple Properties
zfs get propertyA,propertyB,propertyC POOLNAME
zfs get propertyA,propertyB,propertyC POOLNAME/DATASET_NAME

Example of some properties you'd check and the output

# Single Property in a Pool
zfs get compression INTREPID

NAME      PROPERTY     VALUE           SOURCE
INTREPID  compression  zstd-7          local


# Single Property in a Dataset
zfs get compression INTREPID/mirrors/apt

NAME                  PROPERTY     VALUE           SOURCE
INTREPID/mirrors/apt  compression  zstd-12         inherited from INTREPID/mirrors


# Multiple Properties in a Pool
zfs get compression,compressratio,mountpoint,encryption,atime INTREPID

NAME      PROPERTY       VALUE              SOURCE
INTREPID  compression    zstd-7             local
INTREPID  compressratio  1.14x              -
INTREPID  mountpoint     /storage/INTREPID  local
INTREPID  encryption     off                default
INTREPID  atime          off                local

Pool and Dataset mountpoint

When you set this on a Pool or a Dataset, ZFS will automatically mount the Pool or Dataset to this filesystem path. The path is case sensitive and doesn't have to match your POOLNAME.

Mountpoint for a POOL

zfs set mountpoint=/path/to/dir POOLNAME

Mountpoint for a Dataset

zfs set mountpoint=/path/to/dir POOLNAME/DATASET_NAME

Example of my mountpoint

zfs set mountpoint=/storage/ENTERPRISE ENTERPRISE

Compression on a Pool or Dataset

ZFS allows you to define compression globally on a pool and specifically on Datasets. Compression is inherited from the parent filesystem unless it has its own compression set. You can change the compression on a pool or dataset anytime, but the data is not compressed to the new format and level until new data is written.

zfs set compression=OPTION POOLNAME
zfs set compression=OPTION POOLNAME/DATASET_NAME

Here are the options available

on | off | lzjb | gzip | gzip-[1-9] | zle | lz4 | zstd | zstd-[1-19] | zstd-fast | zstd-fast-[1-10,20,30,40,50,60,70,80,90,100,500,1000]

on defaults to lz4. zstd alone means zstandard level 3 when set. But you can specify different zstd levels with zstd-# from compressional level 1 to 19. zstd-fast and its options utilize a faster zstd compression method, but the compression ratio will be less.

My recommendation is to utilize zstandard, specifically zstd-7 as a global option on the Pool, and use zstd-12 for Datasets that can benefit from higher compression and the data write speeds don't matter as much. You can also set zstd-15 or zstd-19 for maximum space savings for backup or archival data.

# Set global zstandard level 7 on POOL globally
zfs set compression=zstd-7 POOLNAME

# Set higher zstandard compression on specific Dataset
zfs set compression=zstd-12 POOLNAME/DATASET_NAME

See what compression is being used on a Pool or Dataset

zfs get compression POOLNAME
zfs get compression POOLNAME/DATASET_NAME

See how much compression is helping - Compression Ratio

zfs get compressratio POOLNAME
zfs get compressratio POOLNAME/DATASET_NAME

Example of the output
The higher the value, the higher % of the data is compressed. 1.82x basically means 82% of the space is saved by compression.

zfs get compressratio INTREPID
NAME      PROPERTY       VALUE  SOURCE
INTREPID  compressratio  1.13x  -

zfs get compressratio INTREPID/containers/mikesulsenti.com
NAME                                  PROPERTY       VALUE  SOURCE
INTREPID/containers/mikesulsenti.com  compressratio  1.82x  -

atime (Access Timestamps)

Disable atime on a Pool or Dataset

zfs set atime=off POOLNAME
zfs set atime=off POOLNAME/DATASET_NAME

Check atime on a Pool or Dataset

zfs get atime POOLNAME
zfs get atime POOLNAME/DATASET_NAME

Dataset and User Disk Space Quotas

ZFS allows you to set disk space quotas on datasets which limits how much space a Dataset can use total or limit how much a Dataset can use in Data and ZFS overheads like snapshots and metadata, or even specific Users or Groups that can use that Dataset filesystem.

These properties accept inputs in Byte units like: T - Terabytes, G - Gigabytes, M - Megabytes, K - Kilobytes

You can check on a Dataset's quota and refquota settings like this:

zfs get refquota,quota POOLNAME/DATASET_NAME

And the available disk space for the Datasets with quotas should show in zfs list

Quota - Limit Total Disk Usage in a Dataset

The ZFS property flag quota sets the hard limit of how much disk space a Dataset can use, including actual data, snapshots, and any other extras. Snapshots will use disk space if there are changes between snapshot(s) and the live data. This caps both.

# Syntax of quota
zfs set quota=[SIZE] POOLNAME/DATASET_NAME

# Real Example setting this is 1,200 Gigabytes max Total Quota
zfs set quota=1200G INTREPID/users/mike

Or remove it with none value

zfs set quota=none INTREPID/users/mike

RefQuota - Limit Data Disk Usage in a Dataset

refquota sets a limit on the live data disk usage of the dataset but does not have a limit on the snapshots and other ZFS data that can add to the total disk usage by the Dataset. You can set both refquota and quota to provide two levels of disk quotas, a hard limit of the Data and a hard limit of the total ZFS data.

# Syntax of refquota
zfs set refquota=[SIZE] POOLNAME/DATASET_NAME

# Real Example setting this is 1,000 Gigabytes max Data Quota
zfs set refquota=1000G INTREPID/users/mike

Now I can also set the quota on this Dataset to a slightly higher number to give some space to potential snapshot utilization even if the data is maxed.

Or remove it with none value

zfs set refquota=none INTREPID/users/mike

User and Group Quotas - Limit Disk Usage by Users and Groups

Similar to how you can set a quota for a Dataset, you can set quotas for specific users or specific groups.

View Disk Usage by User or Group

# View disk usage by Users
zfs userspace POOLNAME/DATASET_NAME

# View disk usage by Groups
zfs groupspace POOLNAME/DATASET_NAME

Example output

zfs userspace INTREPID/users/mike
TYPE        NAME   USED  QUOTA  OBJUSED  OBJQUOTA
POSIX User  mike  1.00T   none    1.64M      none
POSIX User  root     3K   none        6      none

zfs groupspace INTREPID/users/mike
TYPE         NAME   USED  QUOTA  OBJUSED  OBJQUOTA
POSIX Group  mike  1.00T   none    1.64M      none
POSIX Group  root     3K   none        6      none

Set User Quota

# Syntax for setting a user quota
zfs set userquota@USERNAME=[SIZE] POOLNAME/DATASET_NAME

# Real example for setting user 'mike' to 200 Gigabytes max
zfs set userquota@mike=200G INTREPID/shares/Documents

Or remove it with none value

zfs set userquota@mike=none INTREPID/shares/Documents

Set Group Quota

# Syntax for setting a group quota
zfs set groupquota@GROUPNAME=[SIZE] POOLNAME/DATASET_NAME

# Real example for setting group 'accounting' to 500 Gigabytes max
zfs set groupquota@accounting=500G INTREPID/shares/Documents

Or remove it with none value

zfs groupquota@accounting=none INTREPID/shares/Documents

Check Pool Status

zpool status is going to be the way to status of the Pool health and the drives

# Check zpool status of all Pools
zpool status

# Checkk all Pools with extra Verbose information
zpool status -v

# Check a specific Pool (with Verbose info)
zpool status -v POOLNAME

Example output

state shows whether the Pool or Device is online or healthy

READ, WRITE, and CKSUM indicate if there's ERRORs with the data. You don't want to see anything in these columns but 0. But if you see error counts, it could indicate a bad cable or a bad drive.

  pool: ENTERPRISE
 state: ONLINE
  scan: scrub repaired 0B in 09:00:00 with 0 errors on Sun Jan  8 09:24:01 2023
config:

	NAME                        STATE     READ WRITE CKSUM
	ENTERPRISE                  ONLINE       0     0     0
	  raidz1-0                  ONLINE       0     0     0
	    wwn-0x5000cca24cc243a9  ONLINE       0     0     0
	    wwn-0x5000cca24cc3cf60  ONLINE       0     0     0
	    wwn-0x5000cca24cc3c39a  ONLINE       0     0     0
	    wwn-0x5000cca24cc3c304  ONLINE       0     0     0

errors: No known data errors


  pool: INTREPID
 state: ONLINE
  scan: scrub repaired 0B in 00:52:52 with 0 errors on Sun Jan  8 01:16:54 2023
config:

	NAME                                  STATE     READ WRITE CKSUM
	INTREPID                              ONLINE       0     0     0
	  raidz1-0                            ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208190540104  ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208190540533  ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208290490136  ONLINE       0     0     0
	    ata-TEAM_T2532TB_112208190540057  ONLINE       0     0     0

errors: No known data errors

Example of an output after a resilvering (re-syncing data on a replaced or offline drive)

  pool: INTREPID
 state: ONLINE
  scan: resilvered 70.0G in 00:17:40 with 0 errors on Tue Jan 17 12:06:45 2023
config:

	NAME                                       STATE     READ WRITE CKSUM
	INTREPID                                   ONLINE       0     0     0
	  raidz2-0                                 ONLINE       0     0     0
	    ata-TEAM_T2532TB_TPBF2209160130700561  ONLINE       0     0     0
	    ata-TEAM_T2532TB_TPBF2209160130700881  ONLINE       0     0     0
	    ata-TEAM_T2532TB_TPBF2209160130700462  ONLINE       0     0     0
	    ata-TEAM_T2532TB_TPBF2209160130700420  ONLINE       0     0     0

errors: No known data errors

Pool Filesystem Statistics

zpool list outputs information about the Pool filesystem like Total Size, Allocated Usage, Free Space, Fragmented Free Space, Capacity, Deduplication Ratio, and Health status.

You can use zpool list -v to show more detailed information of the Pool broken down by Device and Pool Type.

You can also customize the command to show the properties you want to list out. Use zpool list -h to show what options you can list out.

Example usage and outputs

zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ENTERPRISE  14.5T  7.34T  7.20T        -         -     3%    50%  1.00x    ONLINE  -
INTREPID    7.44T  2.93T  4.51T        -         -     2%    39%  1.00x    ONLINE  -


zpool list -v
NAME                                   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ENTERPRISE                            14.5T  7.34T  7.20T        -         -     3%    50%  1.00x    ONLINE  -
  raidz1-0                            14.5T  7.34T  7.20T        -         -     3%  50.5%      -    ONLINE
    wwn-0x5000cca24cc243a9            3.64T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca24cc3cf60            3.64T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca24cc3c39a            3.64T      -      -        -         -      -      -      -    ONLINE
    wwn-0x5000cca24cc3c304            3.64T      -      -        -         -      -      -      -    ONLINE
INTREPID                              7.44T  2.93T  4.51T        -         -     2%    39%  1.00x    ONLINE  -
  raidz1-0                            7.44T  2.93T  4.51T        -         -     2%  39.4%      -    ONLINE
    ata-TEAM_T2532TB_112208190540104  1.86T      -      -        -         -      -      -      -    ONLINE
    ata-TEAM_T2532TB_112208190540533  1.86T      -      -        -         -      -      -      -    ONLINE
    ata-TEAM_T2532TB_112208290490136  1.86T      -      -        -         -      -      -      -    ONLINE
    ata-TEAM_T2532TB_112208190540057  1.86T      -      -        -         -      -      -      -    ONLINE

Pool IO Stats

zpool iostat provides IO statistics. Passing the verbose option zpool iostat -v provides a greater breakdown of the IO information. You can also target a specific pool with zpool iostat -v POOLNAME

zpool iostat
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
ENTERPRISE  7.34T  7.20T     33     60  5.60M  3.50M
INTREPID    2.93T  4.51T     39    132  2.09M  1.70M
----------  -----  -----  -----  -----  -----  -----


zpool iostat -v
                                        capacity     operations     bandwidth
pool                                  alloc   free   read  write   read  write
------------------------------------  -----  -----  -----  -----  -----  -----
ENTERPRISE                            7.34T  7.20T     33     60  5.60M  3.50M
  raidz1-0                            7.34T  7.20T     33     60  5.60M  3.50M
    wwn-0x5000cca24cc243a9                -      -      8     15  1.41M   902K
    wwn-0x5000cca24cc3cf60                -      -      8     14  1.39M   891K
    wwn-0x5000cca24cc3c39a                -      -      8     15  1.41M   902K
    wwn-0x5000cca24cc3c304                -      -      8     14  1.39M   892K
------------------------------------  -----  -----  -----  -----  -----  -----
INTREPID                              2.93T  4.51T     39    132  2.09M  1.70M
  raidz1-0                            2.93T  4.51T     39    132  2.09M  1.70M
    ata-TEAM_T2532TB_112208190540104      -      -     10     34   535K   438K
    ata-TEAM_T2532TB_112208190540533      -      -      9     32   535K   435K
    ata-TEAM_T2532TB_112208290490136      -      -     10     34   536K   438K
    ata-TEAM_T2532TB_112208190540057      -      -      9     32   535K   434K
------------------------------------  -----  -----  -----  -----  -----  -----

ZFS Pool Scrub

zpool scrub initiates a full Pool data check, while the Pool is still online and usable. It will scan through all the data in the Pool and verify that the data matches the checksum. If it finds an issue, it will attempt to repair that data on the Device with data it knows is good.

Scrubs should be done on a monthly basis. How long a scrub takes to perform depends on if it's being done on spinning Hard Drives or SSDs, how much data is on the Pool, how many drives are in the Pool to speed up read times, and how much IO is using the Pool during the duration. Large SSD arrays can do their scrubs in hours while large (old) HDD arrays I have seen take half a month to almost a month to finish.

Start a scrub manually on a POOL (or resume a paused scrub)

zpool scrub POOLNAME

Pause a scrub in progress (can be resumed later)

zpool scrub -p POOLNAME

Stop/cancel a scrub in progress

zpool scrub -s POOLNAME

Check on the Status of a Scrub (zpool status)

zpool status -v POOLNAME

  ...
  scan: scrub in progress since Sun Jul 25 16:07:49 2021
        403M scanned at 100M/s, 68.4M issued at 10.0M/s, 405M total
        0B repaired, 16.91% done, 00:00:04 to go
  ...

Enable Periodic Scrubs

Proxmox automatically does this, but for other Linux systems, you can enable a systemd timer for this like so:

# Scrub POOLNAME weekly
systemctl enable [email protected] --now

# Scrub POOLNAME Monthly (recommended)
systemctl enable [email protected] --now

Or simply just setup a cronjob that calls the scrub command.

# Scrub all pools on the second Sunday of every month.
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi

# Or Simply on the 15th of every month for a specific Pool
0 2 15 * * zpool scrub POOLNAME >/dev/null 2>&1

ZFS Pool TRIM

While Linux/Systemd has a FSTRIM.timer that is usually enabled by default on most distros, this may not actually TRIM ZFS Pools.

zpool trim POOLNAME

Enable Periodic TRIMs

Proxmox automatically does this, but for other Linux systems, you can enable a systemd timer for this like so:

# TRIM POOLNAME weekly
systemctl enable [email protected] --now

# TRIM POOLNAME monthly (recommended)
systemctl enable [email protected] --now

Or setup a cronjob for it

# TRIM all Pools the first Sunday of every month.
24 0 1-7 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi

# Or Simply on the 5th of every month for a specific Pool
0 2 5 * * zpool trim POOLNAME >/dev/null 2>&1

ZFS ARC Stats and Summary

arcstat and arc_summary can provide useful and advanced information on how ZFS's ARC Cache is being used.

arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
16:11:59     0     0      0     0    0     0    0     0    0  6.6G  8.5G   1.9G

arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Jan 20 16:12:06 2023
Linux 5.15.74-1-pve                                           2.1.6-pve1
Machine: virgo (x86_64)                                       2.1.6-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    43.2 %    6.6 GiB
        Target size (adaptive):                        55.1 %    8.5 GiB
        Min size (hard limit):                          6.2 %  983.5 MiB
        Max size (high water):                           16:1   15.4 GiB
        Most Frequently Used (MFU) cache size:         21.2 %    1.0 GiB
        Most Recently Used (MRU) cache size:           78.8 %    3.8 GiB
        Metadata cache size (hard limit):              75.0 %   11.5 GiB
        Metadata cache size (current):                 24.4 %    2.8 GiB
        Dnode cache size (hard limit):                 10.0 %    1.2 GiB
        Dnode cache size (current):                    90.4 %    1.0 GiB

ARC hash breakdown:
        Elements max:                                             776.7k
        Elements current:                              25.6 %     198.9k
        Collisions:                                                24.9M
        Chain max:                                                     5
        Chains:                                                     4.5k

ARC misc:
        Deleted:                                                  185.8M
        Mutex misses:                                              50.7k
        Eviction skips:                                           325.7k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                  10.1 TiB
        L2 eligible MFU evictions:                     33.2 %    3.3 TiB
        L2 eligible MRU evictions:                     66.8 %    6.7 TiB
        L2 ineligible evictions:                                10.6 TiB

ARC total accesses (hits + misses):                                 3.6G
        Cache hit ratio:                               96.7 %       3.5G
        Cache miss ratio:                               3.3 %     119.7M
        Actual hit ratio (MFU + MRU hits):             96.5 %       3.5G
        Data demand efficiency:                        99.4 %       1.3G
        Data prefetch efficiency:                       1.2 %      86.8M

Cache hits by cache type:
        Most frequently used (MFU):                    86.2 %       3.0G
        Most recently used (MRU):                      13.6 %     475.0M
        Most frequently used (MFU) ghost:               0.1 %       4.7M
        Most recently used (MRU) ghost:               < 0.1 %       1.3M
        Anonymously used:                             < 0.1 %     772.9k

Cache hits by data type:
        Demand data:                                   36.6 %       1.3G
        Prefetch data:                                < 0.1 %    1000.0k
        Demand metadata:                               63.1 %       2.2G
        Prefetch metadata:                              0.2 %       7.7M

Cache misses by data type:
        Demand data:                                    6.3 %       7.5M
        Prefetch data:                                 71.6 %      85.8M
        Demand metadata:                               17.3 %      20.7M
        Prefetch metadata:                              4.8 %       5.8M

DMU prefetch efficiency:                                          366.1M
        Hit ratio:                                     21.6 %      79.1M
        Miss ratio:                                    78.4 %     287.0M

L2ARC not detected, skipping section

### The output continues so I just cut it off here.

ZFS Pool Upgrade

Every so often, a ZFS Utils update introduces new features. ZFS Pools on BSD, Linux, and macOS do not use a version number, but feature flags. When a ZFS Pool Upgrade is available, zpool status -v will show that upgrades are available. Note: If you do the upgrade, the Pool won't be able to be used in an older ZFS system. This usually is not an issue to be concerned about.

You can upgrade the pool with:

# Check all Pools for upgrades
zpool upgrade

# Enable all supported features on all Pools
zpool upgrade -a

# Enable for specific Pool
zpool upgrade -a POOLNAME

Offline Drive

Temporarily Offline a Drive

A Temporarily offline Drive comes back online automatically after a reboot or you put it back Online

zpool offline -t POOLNAME DRIVE_NAME

Example of real world use

zpool offline -t INTREPID ata-TEAM_T2532TB_112208190540533

Offline a Drive permanently (and by force)

The -f flag forces the drive to be offline in a faulted state. This is a step more than a normal Offline as it persistence between Pool imports. Do not use the -f flag unless you need the drive to stay offline between imports.

zpool offline -f POOLNAME DRIVE_NAME

Online a Drive

Brings the physical Drive online

zpool online POOLNAME DRIVE_NAME

Bring the Drive online AND Expand -e the Drive to use available space on Drive. If the Device is part of a mirror or RAID-Z, all Devices must be expanded before te new space is usable on the Pool.

zpool online -e POOLNAME DRIVE_NAME

Replace a Drive

You can replace a drive in a ZFS Pool while the old drive is still in the Pool. The new drive will be added the Pool and resilvered. After this is completed, the OLD Drive will be detached from the Pool. This can be before or after the OLD Drive is removed from the system.

zpool replace POOLNAME OLD_DRIVE_NAME NEW_DRIVE_NAME

Clear Errors

If errors occured on a ZFS Pool, and a solution was applied, you can Clear the errors from the Pool to bring the Pool back from a degraded state.

# Clear All Errors on a Pool
zpool clear POOLNAME

# Specific Drive
zpool clear POOL DRIVE_NAME

Export ZFS Pool

You can export a ZFS Pool to either move it (and the drives) to another system or to utilize proper Drive names, if you accidently made the Pool with the /dev/sda type drive naming convention, that is NOT recommended to have.

zpool export POOLNAME

All Datasets and Pools will be unmounted and the Pool will probably disappear from zpool status

Import ZFS Pool

You an import a ZFS Pool that was exported or came from another system.

List importable Pools. Not all importable Pools may show up this method.

zpool import

  pool: POOLNAME
    id: 15451357997522795478
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.

Then import via the name or ID

zpool import POOLNAME

Import by scanning Drives

If you're unsure about what Drives to import by or there's a lot to import, you can use this to scan through the Disks by ID name and import them. For some HDDs, it may default to using wwn- names, which isn't as good as the actual Drive name.

-a Means that it searches for and imports found Pools

-N Means it Imports the Pool without automounting filesystems

-d Means it searches a directory or a device

zpool import -d /dev/disk/by-id -aN

So you can specify the -d option multiple times to import the Drive name you specifically want:

zpool import -d ata-ST16000NM000J-2TW103_ZR52CQQK -d ata-ST16000NM000J-2TW103_ZR57LBNF -d ata-ST16000NM000J-2TW103_ZR5CZTZ6 -d ata-ST16000NM000J-2TW103_ZR5D4592 -d ata-ST16000NM000J-2TW103_ZR5D4D23 -d ata-ST16000NM000J-2TW103_ZR60QV5P -aN

Import a Pool that was DESTROYED

You can re-import a Pool that was Destroyed with zpool destroy POOLNAME with the -D flag

# Import Destroyed Pool by autoscanning Disk IDs
zpool import -D -d /dev/disk/by-id -aN

# Import Destroyed Pool by specific Drive names
zpool import -D -d ata-ST16000NM000J-2TW103_ZR52CQQK -d ata-ST16000NM000J-2TW103_ZR57LBNF -d ata-ST16000NM000J-2TW103_ZR5CZTZ6 -d ata-ST16000NM000J-2TW103_ZR5D4592 -d ata-ST16000NM000J-2TW103_ZR5D4D23 -d ata-ST16000NM000J-2TW103_ZR60QV5P -aN

ZPOOL ADD

Adding to your ZFS Pool usually means a couple of things. Do not mistake this with "expanding" your Pool.

Mirror

Add a mirror to your Pool. If you are running a ZPOOL Mirror already or a ZFS RAID 10, you can add additional mirrors to your Pool

zpool add POOLNAME mirror DRIVE_NAME_1 DRIVE_NAME_2 ...

Cache Drive - L2ARC

You can add a Cache or L2ARC to your ZFS Pool to speed up Read speeds and latency. Your Cache device should have a faster speed than your actual Pool, so you'll want to utilize SSDs for a HDD Pool, or NVMe SSD for a SATA SSD Pool. You can have multiple Cache drives for the Pool, which allows redundancy. More space though for L2ARC is not usually going to be beneficial, this is after your ARC Cache in memory is used.

zpool add POOLNAME cache DRIVE_NAME [DRIVE_NAME_2 if you use more than 1 drive]

You can remove this Drive from the Pool

zpool remove POOLNAME CACHE_DRIVE_NAME

Log Drive - ZIL - ZFS Intent Log Drive

You can add a ZIL or Intent Log drive to your ZFS Pool. These should ideally be very fast and Cache-less NVMe Drives. Intel Optane drives make the best Log Drives. They only need to be around 16-20GB in size. ZFS will only store about 5 seconds of writes to the drives before committing them to the Pool. If you have SSDs as your Pool storage already, this may not benefit you much if at all.

Additionally you should ideally have TWO Log drives in case one of them fails. A failed Log drive alone is not an issue but a failed Log drive and a power failure of the system could cause data corruption for the data that was about to be written.

# Single Log drive
zpool add POOLNAME log DRIVE_NAME

# Redundent Log drives
zpool add POOLNAME log mirror DRIVE_NAME_1 DRIVE_NAME_2

You can remove the ZIL Log Drive from the Pool

zpool remove POOLNAME LOG_DRIVE_NAME

ZFS Pool Hot Spares

These Drives can be added to your Pool to automatically replace a failed Drive.

zpool add POOLNAME spare DRIVE_NAME

The Spares will be listed in their own category from the active Drives

Remove a spare at anytime (as long as a Drive is not currently failed)

zpool remove POOLNAME SPARE_DRIVE_NAME

ZFS has the ability to Snapshot entire Pools and Datasets. These snapshots take no space except for the change or delta in files from the live filesystem. They can be sent to other Datasets, ZFS Pools, mounted, or rollbacked to.

Create a ZFS Snapshot

ZFS Snapshots are named with the @ symbol first and then a name for that Snapshot. This is case sensitive and it can be whatever you want. Do not use spaces and I would avoid certain special characters. But this could be, but not limited to:

• @today - @yesterday
• @2023-01-15 - @jan-15-2023
• @focal - @bionic
• @3-days-ago - @5daysago
• @v3.22 - @v2.09

# Snapshot a Pool
zfs snapshot POOLNAME@SNAPSHOT_NAME
zfs snapshot POOLNAME/DATASET_NAME@SNAPSHOT_NAME

Recursively Create Snapshots for Dataset and Children Datasets

Useful if you want all the Datasets of the target Dataset to have their own Snapshots at the same time, and for scripting.

-r is the Recursive flag to apply

zfs snapshot -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME

Rollback a Dataset to its last Snapshot

You can easily rollback all changes made to a Dataset's filesystem/files/data to the last Snapshot made of that Dataset.

Note: Rolling back to the last Snapshot will delete all changes made including: new files, modified files, etc

# Syntax of a Snapshot Rollback
zfs rollback POOLNAME/DATASET_NAME@SNAPSHOT_NAME

# Real Example
zfs rollback INTREPID/containers/mikesulsenti.com@2023-01-08

You can only rollback to the latest Snapshot. If you want to rollback to an even earlier Snapshot that still exists, you must either destroy the snapshots between that snapshot and the live Dataset. Or run rollback with the -r option to automatically delete all snapshots inbetween.

zfs rollback -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME

Rename A ZFS Snapshot

Similar to renaming a Dataset, you can rename a Snapshot. useful for scripting or moving things around.

# Change a specific @SNAPSHOT_NAME to a New Name
zfs rename POOLNAME/DATASET_NAME@SNAPSHOT_NAME @NEW_SNAPSHOT_NAME

# Recursively change all @SNAPSHOT_NAME Snapshots to a New Name
zfs rename -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME @NEW_SNAPSHOT_NAME

Destroy or Delete a ZFS Snapshot

Destroys a ZFS Snapshot and removes it and the data from the filesystem. Does not touch the live filesystem/Dataset

# Destroy a specific Dataset's Snapshot
zfs destroy POOLNAME/DATASET_NAME@SNAPSHOT_NAME

# Recursively Destroy @SNAPSHOT_NAME Snapshots
zfs destroy -r POOLNAME/DATASET_NAME@SNAPSHOT_NAME

Clone a Snapshot to it's own Dataset

You can take a Dataset's Snapshot and make it into it's own Dataset and live filesystem

zfs clone POOLNAME/DATASET_NAME@SNAPSHOT_NAME POOLNAME/NEW_DATASET_NAME

You then need to Promote the clone Dataset to no longer depend on the origin Snapshot

zfs promote POOLNAME/NEW_DATASET_NAME

ZFS Send and Receive

-R is Replicate and it replicates the filesystem properties and children filesystems, including all snapshots.

-i is for snapshot and sends incremental changes

-v is verbose output and shows how much data has been sent and more each second

First you need to snapshot the Dataset or Pool. Check the section above on how to do that

zfs snapshot POOLNAME/DATASET_NAME@SNAPSHOT_NAME

ZFS Send and Receive Locally

# Replicate a Snapshot
zfs send -R -v POOLNAME/DATASET_NAME@SNAPSHOT_NAME | zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME

# Replicate a Snapshot with Incremental Changes
zfs send -R -i -v POOLNAME/DATASET_NAME@SNAPSHOT_NAME | zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME

ZFS Send and Receive Remotely

Simply just pipe into the ssh command with the hostname or user@hostname

# Replicate a Snapshot
zfs send -R POOLNAME/DATASET_NAME@SNAPSHOT_NAME | ssh user@hostname zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME

# Replicate a Snapshot with Incremental Changes
zfs send -R -i POOLNAME/DATASET_NAME@SNAPSHOT_NAME | ssh user@hostname zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME

Monitor Progress of a ZFS Send

zfs send has a -v verbose option, but this may produce too much information. You can utilize pv to have a more simplified info, but you may not get a full picture like -v provides.

You can install the utility pv which is Pipe View. If you put pv between the pipes, you can watch the data stream being sent. You can install it on Debian/Ubuntu/Proxmox with apt install pv and on Red Hat/CentOS/Fedora/AlmaLinux/Rocky Linux with yum install pv

# Monitor locally
zfs send -R -i POOLNAME/DATASET_NAME@SNAPSHOT_NAME | pv | zfs receive OTHERPOOLNAME/OTHER_DATASET_NAME

# Monitor Remotely over SSH
zfs send -R -i POOLNAME/DATASET_NAME@SNAPSHOT_NAME | pv | ssh user@hostname zfs receive OTHER_POOLNAME/OTHER_DATASET_NAME

Create an Encrypted Dataset

The important options here are:

• encryption - Options: on, off, aes-256-gcm (default for on currently and recommended), aes-256-ccm (OLD default), aes-192-gcm, aes-192-ccm, aes-128-gcm, aes-128-ccm
• keylocation - Options: prompt (prompts in shell when you tell ZFS to load Dataset), file:///path/to/key_file.txt
• keyformat - Options: passphrase (8-512 bytes long and recommended), hex, raw (hex and raw must be 32 bytes long)

Create an Encrypted Dataset with aes-256-gcm, prompt with command, using passphrase key

zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase POOLNAME/ENCRYPTED_DATASET_NAME

Load the key to mount Dataset after a reboot

-r Recursively loads the key for any child Dataset encrypted with the key. If a Child Dataset has its own key, you will need to target load-key for that Dataset

zfs load-key -r POOLNAME/ENCRYPTED_DATASET_NAME

Or load-key using a key file

zfs load-key -r -L file:///path/to/key_file.txt POOLNAME/ENCRYPTED_DATASET_NAME

You can also specify -a to load that key for any encrypted volume on any active Pool in the system

Change User Encryption Key

You can not change the Master Key to an encrypted Dataset. You'll have to move the data to another new Encrypted Dataset to fulled change out the Master Key. But this changes the user key wrapper.

zfs change-key [-l] [-o keylocation=location] [-o keyformat=format] [-o pbkdf2iters=value] POOLNAME/ENCRYPTED_DATASET_NAME

Sending an Encrypted Dataset

When you send a Dataset with zfs send, if you want that data to be encrypted on the other receiving end, you must specify -w to keep the data encrypted as is. This also means you can not compress the data stream as the encrypted data is incompressible.

zfs send -w -R -i POOLNAME/ENCRYPTED_DATASET_NAME@SNAPSHOT_NAME | zfs  receive OTHER_POOLNAME/OTHER_ENCRYPTED_DATASET_NAME

I may update this guide and cheat sheet with more information later or with any new major changes OpenZFS introduces.

If you see an issue in this post, feel free to contact me about it!