In my homelab I have Dell PowerEdge R620 Server with FreeBSD 14.3 and ZFS 2.2.7. I want to use this server for BHyVe server virtualization and run Virtual Machines on top of BHyVe hypervisor.
In virtualized environment, the typical average I/O size differs based on workload running in virtual machines. Different applications generate distinct I/O patterns.
- Databases and transactional systems: These often produce a large number of small, random I/O requests (e.g., 4KB, 8KB, or 16KB). This is because they frequently read and write small chunks of data to update records, log transactions, and access indexes.
- Virtual Desktop Infrastructure (VDI): VDI workloads are notoriously random and write-heavy, with an average I/O size often falling in the 24KB to 32KB range.
- File servers and data backups: These workloads typically generate large, sequential I/O requests (e.g., 64KB, 128KB, 256KB, or larger) as they read or write large files in a continuous stream.
When I look at a typical enterprise cloud datacenter, where the types of workloads are not under your control, I usually observe the average I/O size between 40 KB/s and 64 KB/s. That's the reason why I typically test 32 KB I/O size, however, if you know the specific type of workload you are interested, you should test application specific I/O size.
I have system with 6x NL-SAS in RAIDZ with 40 GB write-cache (ZFS SLOG on NVMe) and 800 GB read-cache (L2ARC on NVMe).
On such system, I was able to achieve following performance results ...
- 32KB IO, 100% read / 0% write
- 32KB IO, 100% read, 100% random, 1 job
- up to 2,617 READ IOPS (85.8 MB/s) @ ~0.38 ms
- 32KB IO, 100% read, 100% random, 2 job
- up to 5,171 READ IOPS (169 MB/s) @ ~0.384 ms
- 32KB IO, 100% read, 100% random, 4 jobs
- up to 10,400 READ IOPS (342 MB/s) @ ~0.38 ms
- 32KB IO, 100% read, 100% random, 8 jobs
- up to 8,799 READ IOPS (288 MB/s) @ ~0.9 ms
- 32KB IO, 70% read / 30%
- 32KB IO, 70% read / 30% write, 100% random, 1 job
- up to 2,030 READ IOPS (66.5 MB/s) @ ~0.261 ms
- up to 868 WRITE IOPS (28.5 MB/s) @ ~0.532 ms
- 32KB IO, 70% read / 30% write, 100% random, 2 jobs
- up to 1,200 READ IOPS (39.3 MB/s) @ ~0.8 ms
- up to 518 WRITE IOPS (17 MB/s) @ ~1.98 ms
- 32KB IO, 70% read / 30% write, 100% random, 4 jobs
- up to 3,866 READ IOPS (127 MB/s) @ ~0.066 ms
- up to 1,664 WRITE IOPS (54.6 MB/s) @ ~2.193 ms
- 32KB IO, 70% read / 30% write, 100% random, 8 jobs
- up to 3,416 READ IOPS (112 MB/s) @ ~1.48 ms
- up to 1,459 WRITE IOPS (47.8 MB/s) @ ~1.95 ms
- 32KB IO, 100% write
- 32KB IO, 0% read / 100% write, 100% random, 1 job
- up to 473 WRITE IOPS (15.5 MB/s) @ ~2.1 ms
- 32KB IO, 0% read / 100% write, 100% random, 2 job
- up to 1,736 WRITE IOPS (56.9 MB/s) @ ~1.13 ms
- 32KB IO, 0% read / 100% write, 100% random, 4 job
- up to 1,796 WRITE IOPS (58.9 MB/s) @ ~2.19 ms
- 32KB IO, 0% read / 100% write, 100% random, 8 job
- up to 1,508 WRITE IOPS (49.4 MB/s) @ ~5.24 ms
Please note that above synthetic storage test are very basic and real performance results can vary. However, I wanted to have some basic understanding of storage subsystem performance capabilities and use it as baseline for future tests.
My old spinning disk pool (6× SATA), accelerated by NVMe caches, delivers the best performance with 4 concurrent jobs.
For a 32 KB I/O size, I observed the following results ...
- 32 KB IO, 100% read
- 10,400 READ IOPS (342 MB/s) @ ~0.38 ms
- 32KB IO, 70% read / 30%
- 3,866 READ IOPS (127 MB/s) @ ~0.066 ms
- 1,664 WRITE IOPS (54.6 MB/s) @ ~2.193 ms
- 32KB IO, 100% write
- 1,796 WRITE IOPS (58.9 MB/s) @ ~2.19 ms
In the future, I will test storage performance from BHyVe Guest OSes.
Anyway, if you are interested, next chapters dive deeper into storage performance testing details ...
System inventory
The server has following disk layout ...
 |
| Disk Layout |
... with following mount points.
 |
| Mount Points |
In server are following disks
- 1x USB 16 GB (da8, 14 GB available) - used for Operating System Binaries
- EFI Partition [260 MB]
- An EFI (Extensible Firmware Interface) partition, also known as an ESP (EFI System Partition), is a dedicated partition on a storage device used by computers that boot with UEFI (Unified Extensible Firmware Interface) firmware. It stores the bootloaders, device drivers, and other utility files necessary for the UEFI firmware to start an operating system.
- Root File System (/) [14 GB]
- A freebsd-ufs partition is a disk partition that uses the Unix File System (UFS), which is the traditional and native filesystem for the FreeBSD operating system. This is where the main operating system files and application are stored.
- I mount the root file system in read-only mode to minimize writes to the USB flash drive and extend its lifespan.
- User data (/var, /tmp) are stored separately on traditional rotational HDD (da0)
- 1x SAS HDD 146 GB (da0, 137 GB) - used for OS-DATA
- ZFS POOL of single disk [132 GB]
- mount point: /var - It stores variable data that changes frequently during system operation.
- Log Files
- Process and Lock Files
- System Caches
- etc.
- mount point: /tmp - Used for temporary files
- 1x SAS HDD 146 GB (da1, 137 GB) - used for SWAP
- GPT Partition on single disk used for Memory Swap [130 GB]
- 6x NL-SATA HDD 500 GB (da2; da3; da4; da5; da6; da7, 466 GB) - used for STORAGE-DATA
- ZFS POOL of six disks in RAIDZ2 [1.75 TB]
- mount point: /STORAGE-DATA/bhyve-datastore - Datastore for Bhyve images
- 1x NVMe Module 477 GB (nda0) - used for ZFS SLOG and ZFS L2ARC
- GPT Partition - nda0p1 [40 GB] - ZFS SLOG (ZFS write cache)
- GPT Partition - nda0p2 [400 GB] - ZFS L2ARC (ZFS read cache)
- 1x NVMe Module 932 GB (nda1) - used for ZFS SLOG and ZFS L2ARC
- GPT Partition - nda1p1 [40 GB] - ZFS SLOG (ZFS write cache)
- GPT Partition - nda1p2 [800 GB] - ZFS L2ARC (ZFS read cache)
Storage performance tests
Let's test ZFS mountpoint at /STORAGE-DATA/bhyve-datastore on top of 6x NL-SATA HDD 500 GB on RAIDZ2 (4+2) and accelerated by NVMe Modules (40 GB write cache, 1200 GB read cache).
Storage test will use these parameters:
- X jobs x 100 GB file: This indicates X jobs (--numjobs=X) with a 100 GB file size per job (--size=100G).
- Read/Write Ratio:
- 100% Read: This signifies a 0% write and 100% read workload (--rw=randrw --rwmixread=100)
- 70% Reade: This signifies a 30% write and 70% read workload (--rw=randrw --rwmixread=70)
- 0% Read: This signifies a 100% write and 0% read workload (--rw=randrw --rwmixread=0)
- Block Size 32k: This sets the block size to 32 KB (--bs=32k)
- 100% Random Access: This specifies that the I/O operations should be 100% random (--rw=randrw)
Explanation of additional fio command parameters:
- --name=fio-test: A name for the test run.
- --directory=/path/to/your/storage: Specifies the directory where fio will create its test files.
- --rw=randrw: Sets the I/O pattern to random read/write.
- --rwmixread=10: Defines the mix as 10% reads and 90% writes.
- --bs=32k: Sets the block size to 32 kilobytes.
- --size=20G: Specifies the size of the test file for each job.
- --numjobs=8: Runs 8 parallel test processes.
- --end_fsync=1: The fio parameter --end_fsync=1 forces a file system sync operation after the entire write workload has been completed. By adding this parameter, you're telling fio that after all the writes in the job are done, it must call the fsync(2) system call. The fsync() call forces all the dirty data from the in-memory cache to be physically written to the storage media.
- --iodepth=30: The number of I/O requests to keep in the queue. A higher value can help saturate the storage.
- --direct=1: Bypasses the operating system's page cache, providing a more accurate measure of the storage device's raw performance.
- --runtime=60: Specifies the duration of the test in seconds. A 1-minute run is a short but good starting point.
- --group_reporting: Reports statistics for all jobs in a single summary.
fio installation
mount -uw /
pkg install -y fio
Now we can run particular storage performance tests and during the test we can monitor physical disk activity by command
Performance monitoring
We can monitor physical disks with utility gstat which print statistics about GEOM disks.
gstat -p -I 3s
dT: 3.026s w: 3.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.000 0 0 0.000 0.0 | nda0
0 2417 0 0 0.000 2417 308363 0.710 19.4| nda1
0 0 0 0 0.000 0 0 0.000 0.0| da0
0 0 0 0 0.000 0 0 0.000 0.0| da1
17 105 0 0 0.000 105 103148 118.9 100.4| da2
19 109 0 0 0.000 109 107737 134.1 100.5| da3
10 108 0 0 0.000 108 105611 176.7 99.9| da4
13 112 0 0 0.000 112 102640 165.6 99.7| da5
19 110 0 0 0.000 110 105136 164.5 100.2| da6
12 106 0 0 0.000 106 99394 179.9 99.8| da7
0 0 0 0 0.000 0 0 0.000 0.0| da8
In example above, we can see that NL-SAS disks are handling ~100 IOPS and NVMe read-cache (nda1) is handling ~2400 IOPS (308 MB/s).
We can also monitor ZFS performance with zpool iostat. This is the most essential tool for monitoring ZFS I/O performance. It reports real-time statistics for ZFS pools and devices. Below is example of zfs iostat monitoring with refresh every 5 seconds.
root@bhyve01:/STORAGE-DATA/bhyve-datastore #zpool iostat 5capacity operations bandwidth pool alloc free read write read write ------------ ----- ----- ----- ----- ----- ----- OS-DATA 74.7M 136G 0 0 58 2.81K STORAGE-DATA 1.02T 1.70T 4 208 140K 15.4M ------------ ----- ----- ----- ----- ----- ----- OS-DATA 74.9M 136G 0 14 0 45.1K STORAGE-DATA 1.03T 1.69T 0 2.31K 0 528M ------------ ----- ----- ----- ----- ----- ----- ^C root@bhyve01:/STORAGE-DATA/bhyve-datastore #
zpool iostat -v: Shows statistics for individual vdevs and the underlying physical disks. Below is example of zfs iostat -v monitoring with refresh every 5 seconds.
root@bhyve01:/STORAGE-DATA/bhyve-datastore #zpool iostat -v 5capacity operations bandwidth pool alloc free read write read write ------------ ----- ----- ----- ----- ----- ----- OS-DATA 74.8M 136G 0 0 58 2.83K da0 74.8M 136G 0 0 58 2.83K ------------ ----- ----- ----- ----- ----- ----- STORAGE-DATA 1.12T 1.60T 4 225 139K 16.6M raidz2-0 1.12T 1.60T 4 223 139K 16.3M da2 - - 0 37 22.2K 2.72M da3 - - 0 37 22.3K 2.72M da4 - - 0 36 25.6K 2.72M da5 - - 0 37 25.4K 2.72M da6 - - 0 37 21.9K 2.72M da7 - - 0 36 22.0K 2.72M logs - - - - - - nda1p1 128K 39.5G 0 2 5 282K cache - - - - - - nda1p2 457G 343G 14 48 1.76M 6.07M ------------ ----- ----- ----- ----- ----- ----- ^C root@bhyve01:/STORAGE-DATA/bhyve-datastore #
32 KB IO, 100% read
Storage Performance test - 32 KB IO, 100% read, 100% random, 1 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.40
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3616KiB/s][r=113 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=1): err= 0: pid=5040: Thu Aug 21 20:04:34 2025
read: IOPS=2617, BW=81.8MiB/s (85.8MB/s)(4909MiB/60014msec)
clat (usec): min=7, max=23215, avg=380.29, stdev=2094.32
lat (usec): min=7, max=23215, avg=380.42, stdev=2094.36
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 37],
| 70.00th=[ 37], 80.00th=[ 38], 90.00th=[ 38], 95.00th=[ 40],
| 99.00th=[13435], 99.50th=[14877], 99.90th=[17695], 99.95th=[18482],
| 99.99th=[19530]
bw ( KiB/s): min= 2880, max=859648, per=100.00%, avg=84444.85, stdev=247804.37, samples=119
iops : min= 90, max=26864, avg=2638.88, stdev=7743.89, samples=119
lat (usec) : 10=2.86%, 20=0.21%, 50=92.29%, 100=1.41%, 250=0.02%
lat (usec) : 500=0.25%, 750=0.23%, 1000=0.02%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.40%, 20=2.30%, 50=0.01%
cpu : usr=0.68%, sys=9.45%, ctx=5125, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=157073,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=81.8MiB/s (85.8MB/s), 81.8MiB/s-81.8MiB/s (85.8MB/s-85.8MB/s), io=4909MiB (5147MB), run=60014-60014msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.40
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3616KiB/s][r=113 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=1): err= 0: pid=5040: Thu Aug 21 20:04:34 2025
read: IOPS=2617, BW=81.8MiB/s (85.8MB/s)(4909MiB/60014msec)
clat (usec): min=7, max=23215, avg=380.29, stdev=2094.32
lat (usec): min=7, max=23215, avg=380.42, stdev=2094.36
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 37],
| 70.00th=[ 37], 80.00th=[ 38], 90.00th=[ 38], 95.00th=[ 40],
| 99.00th=[13435], 99.50th=[14877], 99.90th=[17695], 99.95th=[18482],
| 99.99th=[19530]
bw ( KiB/s): min= 2880, max=859648, per=100.00%, avg=84444.85, stdev=247804.37, samples=119
iops : min= 90, max=26864, avg=2638.88, stdev=7743.89, samples=119
lat (usec) : 10=2.86%, 20=0.21%, 50=92.29%, 100=1.41%, 250=0.02%
lat (usec) : 500=0.25%, 750=0.23%, 1000=0.02%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.40%, 20=2.30%, 50=0.01%
cpu : usr=0.68%, sys=9.45%, ctx=5125, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=157073,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=81.8MiB/s (85.8MB/s), 81.8MiB/s-81.8MiB/s (85.8MB/s-85.8MB/s), io=4909MiB (5147MB), run=60014-60014msec
root@bhyve01:~ #
Result:
- Read IOPS: 2617
- Read Throughput: 85.8 MB/s
- Read Average Latency: ~0.38 ms
- Write IOPS: 0
- Write Average Latency: N/A
- Write Throughput: 0 MB/s
Storage Performance test - 32 KB IO, 100% read, 100% random, 2 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=61.6MiB/s][r=1972 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=2): err= 0: pid=5046: Thu Aug 21 20:11:23 2025
read: IOPS=5171, BW=162MiB/s (169MB/s)(9700MiB/60023msec)
clat (usec): min=7, max=396161, avg=384.60, stdev=2164.38
lat (usec): min=7, max=396161, avg=384.76, stdev=2164.39
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 55],
| 70.00th=[ 359], 80.00th=[ 420], 90.00th=[ 515], 95.00th=[ 529],
| 99.00th=[11338], 99.50th=[14222], 99.90th=[23725], 99.95th=[30802],
| 99.99th=[42206]
bw ( KiB/s): min=29632, max=958976, per=100.00%, avg=165697.95, stdev=178606.03, samples=239
iops : min= 926, max=29968, avg=5178.05, stdev=5581.44, samples=239
lat (usec) : 10=1.59%, 20=1.09%, 50=56.65%, 100=0.84%, 250=0.01%
lat (usec) : 500=28.36%, 750=9.86%, 1000=0.22%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.17%, 20=1.03%, 50=0.14%
lat (msec) : 100=0.01%, 500=0.01%
cpu : usr=0.86%, sys=8.29%, ctx=123762, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=310386,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=162MiB/s (169MB/s), 162MiB/s-162MiB/s (169MB/s-169MB/s), io=9700MiB (10.2GB), run=60023-60023msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=61.6MiB/s][r=1972 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=2): err= 0: pid=5046: Thu Aug 21 20:11:23 2025
read: IOPS=5171, BW=162MiB/s (169MB/s)(9700MiB/60023msec)
clat (usec): min=7, max=396161, avg=384.60, stdev=2164.38
lat (usec): min=7, max=396161, avg=384.76, stdev=2164.39
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 55],
| 70.00th=[ 359], 80.00th=[ 420], 90.00th=[ 515], 95.00th=[ 529],
| 99.00th=[11338], 99.50th=[14222], 99.90th=[23725], 99.95th=[30802],
| 99.99th=[42206]
bw ( KiB/s): min=29632, max=958976, per=100.00%, avg=165697.95, stdev=178606.03, samples=239
iops : min= 926, max=29968, avg=5178.05, stdev=5581.44, samples=239
lat (usec) : 10=1.59%, 20=1.09%, 50=56.65%, 100=0.84%, 250=0.01%
lat (usec) : 500=28.36%, 750=9.86%, 1000=0.22%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.17%, 20=1.03%, 50=0.14%
lat (msec) : 100=0.01%, 500=0.01%
cpu : usr=0.86%, sys=8.29%, ctx=123762, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=310386,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=162MiB/s (169MB/s), 162MiB/s-162MiB/s (169MB/s-169MB/s), io=9700MiB (10.2GB), run=60023-60023msec
root@bhyve01:~ #
Result:
- Read IOPS: 5171
- Read Throughput: 169 MB/s
- Read Average Latency: ~0.384 ms
- Write IOPS: 0
- Write Average Latency: N/A
- Write Throughput: 0 MB/s
Storage Performance test - 32 KB IO, 100% read, 100% random, 4 parallel jobs
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=100 \
> --numjobs=4 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=173MiB/s][r=5531 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=4): err= 0: pid=5061: Thu Aug 21 20:17:58 2025
read: IOPS=10.4k, BW=326MiB/s (342MB/s)(19.1GiB/60010msec)
clat (usec): min=7, max=426144, avg=380.74, stdev=1374.35
lat (usec): min=7, max=426144, avg=380.90, stdev=1374.36
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 38],
| 30.00th=[ 39], 40.00th=[ 41], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 570], 80.00th=[ 717], 90.00th=[ 857], 95.00th=[ 955],
| 99.00th=[ 1336], 99.50th=[11076], 99.90th=[15926], 99.95th=[17695],
| 99.99th=[29230]
bw ( KiB/s): min=107136, max=2925888, per=100.00%, avg=335662.90, stdev=158366.16, samples=476
iops : min= 3348, max=91436, avg=10489.40, stdev=4948.96, samples=476
lat (usec) : 10=0.53%, 20=0.64%, 50=58.20%, 100=4.33%, 250=0.17%
lat (usec) : 500=3.00%, 750=15.72%, 1000=13.78%
lat (msec) : 2=2.97%, 4=0.01%, 10=0.09%, 20=0.54%, 50=0.02%
lat (msec) : 100=0.01%, 500=0.01%
cpu : usr=0.76%, sys=9.34%, ctx=227639, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=626861,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=326MiB/s (342MB/s), 326MiB/s-326MiB/s (342MB/s-342MB/s), io=19.1GiB (20.5GB), run=60010-60010msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=100 \
> --numjobs=4 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=173MiB/s][r=5531 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=4): err= 0: pid=5061: Thu Aug 21 20:17:58 2025
read: IOPS=10.4k, BW=326MiB/s (342MB/s)(19.1GiB/60010msec)
clat (usec): min=7, max=426144, avg=380.74, stdev=1374.35
lat (usec): min=7, max=426144, avg=380.90, stdev=1374.36
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 38],
| 30.00th=[ 39], 40.00th=[ 41], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 570], 80.00th=[ 717], 90.00th=[ 857], 95.00th=[ 955],
| 99.00th=[ 1336], 99.50th=[11076], 99.90th=[15926], 99.95th=[17695],
| 99.99th=[29230]
bw ( KiB/s): min=107136, max=2925888, per=100.00%, avg=335662.90, stdev=158366.16, samples=476
iops : min= 3348, max=91436, avg=10489.40, stdev=4948.96, samples=476
lat (usec) : 10=0.53%, 20=0.64%, 50=58.20%, 100=4.33%, 250=0.17%
lat (usec) : 500=3.00%, 750=15.72%, 1000=13.78%
lat (msec) : 2=2.97%, 4=0.01%, 10=0.09%, 20=0.54%, 50=0.02%
lat (msec) : 100=0.01%, 500=0.01%
cpu : usr=0.76%, sys=9.34%, ctx=227639, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=626861,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=326MiB/s (342MB/s), 326MiB/s-326MiB/s (342MB/s-342MB/s), io=19.1GiB (20.5GB), run=60010-60010msec
root@bhyve01:~ #
Result:
- Read IOPS: 10,400
- Read Throughput: 342 MB/s
- Read Average Latency: ~0.380 ms
- Write IOPS: 0
- Write Throughput: 0 MB/s
- Write Average Latency: N/A
Storage Performance test - 32 KB IO, 100% read, 100% random, 8 parallel jobs
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=100 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=100 \
> --numjobs=8 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 8 processes
Jobs: 8 (f=8): [r(8)][100.0%][r=168MiB/s][r=5365 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=8): err= 0: pid=5076: Thu Aug 21 20:23:45 2025
read: IOPS=8799, BW=275MiB/s (288MB/s)(16.1GiB/60035msec)
clat (usec): min=7, max=386034, avg=906.52, stdev=3915.92
lat (usec): min=7, max=386034, avg=906.69, stdev=3915.93
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37],
| 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 498], 60.00th=[ 799],
| 70.00th=[ 996], 80.00th=[ 1188], 90.00th=[ 1418], 95.00th=[ 1614],
| 99.00th=[ 12911], 99.50th=[ 27919], 99.90th=[ 55313], 99.95th=[ 67634],
| 99.99th=[108528]
bw ( KiB/s): min=83776, max=2826432, per=100.00%, avg=281676.44, stdev=95293.56, samples=956
iops : min= 2618, max=88326, avg=8802.41, stdev=2977.92, samples=956
lat (usec) : 10=0.75%, 20=0.37%, 50=47.38%, 100=0.29%, 250=0.01%
lat (usec) : 500=1.24%, 750=7.62%, 1000=12.49%
lat (msec) : 2=28.42%, 4=0.33%, 10=0.03%, 20=0.34%, 50=0.61%
lat (msec) : 100=0.12%, 250=0.01%, 500=0.01%
cpu : usr=0.38%, sys=3.54%, ctx=272757, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=528260,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=16.1GiB (17.3GB), run=60035-60035msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=100 \
> --numjobs=8 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 8 processes
Jobs: 8 (f=8): [r(8)][100.0%][r=168MiB/s][r=5365 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=8): err= 0: pid=5076: Thu Aug 21 20:23:45 2025
read: IOPS=8799, BW=275MiB/s (288MB/s)(16.1GiB/60035msec)
clat (usec): min=7, max=386034, avg=906.52, stdev=3915.92
lat (usec): min=7, max=386034, avg=906.69, stdev=3915.93
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37],
| 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 498], 60.00th=[ 799],
| 70.00th=[ 996], 80.00th=[ 1188], 90.00th=[ 1418], 95.00th=[ 1614],
| 99.00th=[ 12911], 99.50th=[ 27919], 99.90th=[ 55313], 99.95th=[ 67634],
| 99.99th=[108528]
bw ( KiB/s): min=83776, max=2826432, per=100.00%, avg=281676.44, stdev=95293.56, samples=956
iops : min= 2618, max=88326, avg=8802.41, stdev=2977.92, samples=956
lat (usec) : 10=0.75%, 20=0.37%, 50=47.38%, 100=0.29%, 250=0.01%
lat (usec) : 500=1.24%, 750=7.62%, 1000=12.49%
lat (msec) : 2=28.42%, 4=0.33%, 10=0.03%, 20=0.34%, 50=0.61%
lat (msec) : 100=0.12%, 250=0.01%, 500=0.01%
cpu : usr=0.38%, sys=3.54%, ctx=272757, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=528260,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=16.1GiB (17.3GB), run=60035-60035msec
root@bhyve01:~ #
Result:
- Read IOPS: 8,799
- Read Throughput: 288 MB/s
- Read Average Latency: ~0.9 ms
- Write IOPS: 0
- Write Throughput: 0 MB/s
- Write Average Latency: N/A
32 KB IO, 70% read / 30% write
Storage Performance test - 32 KB IO, 70% read / 30% write, 100% random, 1 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=70 \
> --numjobs=1 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.40
Starting 1 process
Jobs: 1 (f=1): [m(1)][5.0%][r=504MiB/s,w=213MiB/s][r=16.1k,w=6824 IOPS][eta 00m:Jobs: 1 (f=1): [m(1)][6.7%][r=331MiB/s,w=145MiB/s][r=10.6k,w=4645 IOPS][eta 00m:Jobs: 1 (f=1): [m(1)][8.3%][r=265MiB/s,w=108MiB/s][r=8493,w=3461 IOPS][eta 00m:5Jobs: 1 (f=1): [m(1)][10.0%][r=187MiB/s,w=78.3MiB/s][r=5968,w=2506 IOPS][eta 00mJobs: 1 (f=1): [m(1)][11.7%][r=138MiB/s,w=60.5MiB/s][r=4416,w=1935 IOPS][eta 00mJobs: 1 (f=1): [m(1)][13.3%][r=134MiB/s,w=57.4MiB/s][r=4273,w=1836 IOPS][eta 00mJobs: 1 (f=1): [m(1)][15.0%][r=125MiB/s,w=56.5MiB/s][r=3985,w=1807 IOPS][eta 00mJobs: 1 (f=1): [m(1)][16.7%][r=129MiB/s,w=54.4MiB/s][r=4140,w=1740 IOPS][eta 00mJobs: 1 (f=1): [m(1)][18.3%][r=125MiB/s,w=54.3MiB/s][r=4008,w=1739 IOPS][eta 00mJobs: 1 (f=1): [m(1)][20.0%][r=124MiB/s,w=52.7MiB/s][r=3971,w=1686 IOPS][eta 00mJobs: 1 (f=1): [m(1)][21.7%][r=124MiB/s,w=51.6MiB/s][r=3954,w=1652 IOPS][eta 00mJobs: 1 (f=1): [m(1)][23.3%][r=126MiB/s,w=52.3MiB/s][r=4038,w=1673 IOPS][eta 00mJobs: 1 (f=1): [m(1)][25.0%][r=117MiB/s,w=53.3MiB/s][r=3755,w=1707 IOPS][eta 00mJobs: 1 (f=1): [m(1)][26.7%][r=122MiB/s,w=52.7MiB/s][r=3888,w=1687 IOPS][eta 00mJobs: 1 (f=1): [m(1)][28.3%][r=122MiB/s,w=52.6MiB/s][r=3896,w=1681 IOPS][eta 00mJobs: 1 (f=1): [m(1)][30.0%][r=126MiB/s,w=51.8MiB/s][r=4047,w=1657 IOPS][eta 00mJobs: 1 (f=1): [m(1)][31.7%][r=87.6MiB/s,w=37.7MiB/s][r=2804,w=1207 IOPS][eta 00Jobs: 1 (f=1): [m(1)][33.3%][r=288KiB/s,w=96KiB/s][r=9,w=3 IOPS][eta 00m:40s] Jobs: 1 (f=1): [m(1)][61.7%][r=1984KiB/s,w=1088KiB/s][r=62,w=34 IOPS][eta 00m:23Jobs: 1 (f=1): [m(1)][63.3%][r=2400KiB/s,w=832KiB/s][r=75,w=26 IOPS][eta 00m:22sJobs: 1 (f=1): [m(1)][65.0%][r=2178KiB/s,w=800KiB/s][r=68,w=25 IOPS][eta 00m:21sJobs: 1 (f=1): [m(1)][66.7%][r=2720KiB/s,w=864KiB/s][r=85,w=27 IOPS][eta 00m:20sJobs: 1 (f=1): [m(1)][68.3%][r=2016KiB/s,w=1088KiB/s][r=63,w=34 IOPS][eta 00m:19Jobs: 1 (f=1): [m(1)][70.0%][r=1697KiB/s,w=768KiB/s][r=53,w=24 IOPS][eta 00m:18sJobs: 1 (f=1): [m(1)][71.7%][r=1024KiB/s,w=416KiB/s][r=32,w=13 IOPS][eta 00m:17sJobs: 1 (f=1): [m(1)][73.3%][r=2720KiB/s,w=1216KiB/s][r=85,w=38 IOPS][eta 00m:16Jobs: 1 (f=1): [m(1)][75.0%][r=2720KiB/s,w=1056KiB/s][r=85,w=33 IOPS][eta 00m:15Jobs: 1 (f=1): [m(1)][76.7%][r=2402KiB/s,w=1121KiB/s][r=75,w=35 IOPS][eta 00m:14Jobs: 1 (f=1): [m(1)][78.3%][r=2144KiB/s,w=832KiB/s][r=67,w=26 IOPS][eta 00m:13sJobs: 1 (f=1): [m(1)][80.0%][r=2304KiB/s,w=864KiB/s][r=72,w=27 IOPS][eta 00m:12sJobs: 1 (f=1): [m(1)][81.7%][r=2818KiB/s,w=1249KiB/s][r=88,w=39 IOPS][eta 00m:11Jobs: 1 (f=1): [m(1)][83.3%][r=2176KiB/s,w=1152KiB/s][r=68,w=36 IOPS][eta 00m:10Jobs: 1 (f=1): [m(1)][85.0%][r=2176KiB/s,w=896KiB/s][r=68,w=28 IOPS][eta 00m:09sJobs: 1 (f=1): [m(1)][86.7%][r=2016KiB/s,w=704KiB/s][r=63,w=22 IOPS][eta 00m:08sJobs: 1 (f=1): [m(1)][88.3%][r=1153KiB/s,w=800KiB/s][r=36,w=25 IOPS][eta 00m:07sJobs: 1 (f=1): [m(1)][90.0%][r=1952KiB/s,w=960KiB/s][r=61,w=30 IOPS][eta 00m:06sJobs: 1 (f=1): [m(1)][91.7%][r=2016KiB/s,w=1536KiB/s][r=63,w=48 IOPS][eta 00m:05Jobs: 1 (f=1): [m(1)][93.3%][r=2050KiB/s,w=800KiB/s][r=64,w=25 IOPS][eta 00m:04sJobs: 1 (f=1): [m(1)][95.0%][r=1760KiB/s,w=1024KiB/s][r=55,w=32 IOPS][eta 00m:03Jobs: 1 (f=1): [m(1)][96.7%][r=2592KiB/s,w=832KiB/s][r=81,w=26 IOPS][eta 00m:02sJobs: 1 (f=1): [m(1)][98.3%][r=1440KiB/s,w=992KiB/s][r=45,w=31 IOPS][eta 00m:01sJobs: 1 (f=1): [m(1)][100.0%][r=2176KiB/s,w=1216KiB/s][r=68,w=38 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=1): err= 0: pid=8831: Sat Aug 23 05:41:48 2025
read: IOPS=2030, BW=63.4MiB/s (66.5MB/s)(3807MiB/60010msec)
clat (usec): min=7, max=548030, avg=261.82, stdev=4138.59
lat (usec): min=7, max=548030, avg=261.98, stdev=4138.61
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 31], 10.00th=[ 32], 20.00th=[ 36],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 38],
| 70.00th=[ 38], 80.00th=[ 38], 90.00th=[ 40], 95.00th=[ 47],
| 99.00th=[ 5735], 99.50th=[ 13173], 99.90th=[ 45351], 99.95th=[ 71828],
| 99.99th=[156238]
bw ( KiB/s): min= 64, max=533632, per=100.00%, avg=65499.36, stdev=122057.65, samples=119
iops : min= 2, max=16676, avg=2046.85, stdev=3814.30, samples=119
write: IOPS=868, BW=27.1MiB/s (28.5MB/s)(1629MiB/60010msec); 0 zone resets
clat (usec): min=11, max=387759, avg=531.82, stdev=4690.51
lat (usec): min=11, max=387760, avg=532.33, stdev=4690.53
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 44],
| 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 186], 60.00th=[ 433],
| 70.00th=[ 482], 80.00th=[ 502], 90.00th=[ 510], 95.00th=[ 515],
| 99.00th=[ 9372], 99.50th=[ 13698], 99.90th=[ 60031], 99.95th=[ 91751],
| 99.99th=[181404]
bw ( KiB/s): min= 64, max=227904, per=100.00%, avg=28748.66, stdev=52576.81, samples=116
iops : min= 2, max= 7122, avg=898.39, stdev=1643.03, samples=116
lat (usec) : 10=2.01%, 20=1.00%, 50=77.11%, 100=2.17%, 250=2.35%
lat (usec) : 500=7.73%, 750=6.04%, 1000=0.01%
lat (msec) : 2=0.55%, 4=0.01%, 10=0.13%, 20=0.72%, 50=0.07%
lat (msec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.82%, sys=12.38%, ctx=30577, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=121824,52127,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=63.4MiB/s (66.5MB/s), 63.4MiB/s-63.4MiB/s (66.5MB/s-66.5MB/s), io=3807MiB (3992MB), run=60010-60010msec
WRITE: bw=27.1MiB/s (28.5MB/s), 27.1MiB/s-27.1MiB/s (28.5MB/s-28.5MB/s), io=1629MiB (1708MB), run=60010-60010msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=70 \
> --numjobs=1 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.40
Starting 1 process
Jobs: 1 (f=1): [m(1)][5.0%][r=504MiB/s,w=213MiB/s][r=16.1k,w=6824 IOPS][eta 00m:Jobs: 1 (f=1): [m(1)][6.7%][r=331MiB/s,w=145MiB/s][r=10.6k,w=4645 IOPS][eta 00m:Jobs: 1 (f=1): [m(1)][8.3%][r=265MiB/s,w=108MiB/s][r=8493,w=3461 IOPS][eta 00m:5Jobs: 1 (f=1): [m(1)][10.0%][r=187MiB/s,w=78.3MiB/s][r=5968,w=2506 IOPS][eta 00mJobs: 1 (f=1): [m(1)][11.7%][r=138MiB/s,w=60.5MiB/s][r=4416,w=1935 IOPS][eta 00mJobs: 1 (f=1): [m(1)][13.3%][r=134MiB/s,w=57.4MiB/s][r=4273,w=1836 IOPS][eta 00mJobs: 1 (f=1): [m(1)][15.0%][r=125MiB/s,w=56.5MiB/s][r=3985,w=1807 IOPS][eta 00mJobs: 1 (f=1): [m(1)][16.7%][r=129MiB/s,w=54.4MiB/s][r=4140,w=1740 IOPS][eta 00mJobs: 1 (f=1): [m(1)][18.3%][r=125MiB/s,w=54.3MiB/s][r=4008,w=1739 IOPS][eta 00mJobs: 1 (f=1): [m(1)][20.0%][r=124MiB/s,w=52.7MiB/s][r=3971,w=1686 IOPS][eta 00mJobs: 1 (f=1): [m(1)][21.7%][r=124MiB/s,w=51.6MiB/s][r=3954,w=1652 IOPS][eta 00mJobs: 1 (f=1): [m(1)][23.3%][r=126MiB/s,w=52.3MiB/s][r=4038,w=1673 IOPS][eta 00mJobs: 1 (f=1): [m(1)][25.0%][r=117MiB/s,w=53.3MiB/s][r=3755,w=1707 IOPS][eta 00mJobs: 1 (f=1): [m(1)][26.7%][r=122MiB/s,w=52.7MiB/s][r=3888,w=1687 IOPS][eta 00mJobs: 1 (f=1): [m(1)][28.3%][r=122MiB/s,w=52.6MiB/s][r=3896,w=1681 IOPS][eta 00mJobs: 1 (f=1): [m(1)][30.0%][r=126MiB/s,w=51.8MiB/s][r=4047,w=1657 IOPS][eta 00mJobs: 1 (f=1): [m(1)][31.7%][r=87.6MiB/s,w=37.7MiB/s][r=2804,w=1207 IOPS][eta 00Jobs: 1 (f=1): [m(1)][33.3%][r=288KiB/s,w=96KiB/s][r=9,w=3 IOPS][eta 00m:40s] Jobs: 1 (f=1): [m(1)][61.7%][r=1984KiB/s,w=1088KiB/s][r=62,w=34 IOPS][eta 00m:23Jobs: 1 (f=1): [m(1)][63.3%][r=2400KiB/s,w=832KiB/s][r=75,w=26 IOPS][eta 00m:22sJobs: 1 (f=1): [m(1)][65.0%][r=2178KiB/s,w=800KiB/s][r=68,w=25 IOPS][eta 00m:21sJobs: 1 (f=1): [m(1)][66.7%][r=2720KiB/s,w=864KiB/s][r=85,w=27 IOPS][eta 00m:20sJobs: 1 (f=1): [m(1)][68.3%][r=2016KiB/s,w=1088KiB/s][r=63,w=34 IOPS][eta 00m:19Jobs: 1 (f=1): [m(1)][70.0%][r=1697KiB/s,w=768KiB/s][r=53,w=24 IOPS][eta 00m:18sJobs: 1 (f=1): [m(1)][71.7%][r=1024KiB/s,w=416KiB/s][r=32,w=13 IOPS][eta 00m:17sJobs: 1 (f=1): [m(1)][73.3%][r=2720KiB/s,w=1216KiB/s][r=85,w=38 IOPS][eta 00m:16Jobs: 1 (f=1): [m(1)][75.0%][r=2720KiB/s,w=1056KiB/s][r=85,w=33 IOPS][eta 00m:15Jobs: 1 (f=1): [m(1)][76.7%][r=2402KiB/s,w=1121KiB/s][r=75,w=35 IOPS][eta 00m:14Jobs: 1 (f=1): [m(1)][78.3%][r=2144KiB/s,w=832KiB/s][r=67,w=26 IOPS][eta 00m:13sJobs: 1 (f=1): [m(1)][80.0%][r=2304KiB/s,w=864KiB/s][r=72,w=27 IOPS][eta 00m:12sJobs: 1 (f=1): [m(1)][81.7%][r=2818KiB/s,w=1249KiB/s][r=88,w=39 IOPS][eta 00m:11Jobs: 1 (f=1): [m(1)][83.3%][r=2176KiB/s,w=1152KiB/s][r=68,w=36 IOPS][eta 00m:10Jobs: 1 (f=1): [m(1)][85.0%][r=2176KiB/s,w=896KiB/s][r=68,w=28 IOPS][eta 00m:09sJobs: 1 (f=1): [m(1)][86.7%][r=2016KiB/s,w=704KiB/s][r=63,w=22 IOPS][eta 00m:08sJobs: 1 (f=1): [m(1)][88.3%][r=1153KiB/s,w=800KiB/s][r=36,w=25 IOPS][eta 00m:07sJobs: 1 (f=1): [m(1)][90.0%][r=1952KiB/s,w=960KiB/s][r=61,w=30 IOPS][eta 00m:06sJobs: 1 (f=1): [m(1)][91.7%][r=2016KiB/s,w=1536KiB/s][r=63,w=48 IOPS][eta 00m:05Jobs: 1 (f=1): [m(1)][93.3%][r=2050KiB/s,w=800KiB/s][r=64,w=25 IOPS][eta 00m:04sJobs: 1 (f=1): [m(1)][95.0%][r=1760KiB/s,w=1024KiB/s][r=55,w=32 IOPS][eta 00m:03Jobs: 1 (f=1): [m(1)][96.7%][r=2592KiB/s,w=832KiB/s][r=81,w=26 IOPS][eta 00m:02sJobs: 1 (f=1): [m(1)][98.3%][r=1440KiB/s,w=992KiB/s][r=45,w=31 IOPS][eta 00m:01sJobs: 1 (f=1): [m(1)][100.0%][r=2176KiB/s,w=1216KiB/s][r=68,w=38 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=1): err= 0: pid=8831: Sat Aug 23 05:41:48 2025
read: IOPS=2030, BW=63.4MiB/s (66.5MB/s)(3807MiB/60010msec)
clat (usec): min=7, max=548030, avg=261.82, stdev=4138.59
lat (usec): min=7, max=548030, avg=261.98, stdev=4138.61
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 31], 10.00th=[ 32], 20.00th=[ 36],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 38],
| 70.00th=[ 38], 80.00th=[ 38], 90.00th=[ 40], 95.00th=[ 47],
| 99.00th=[ 5735], 99.50th=[ 13173], 99.90th=[ 45351], 99.95th=[ 71828],
| 99.99th=[156238]
bw ( KiB/s): min= 64, max=533632, per=100.00%, avg=65499.36, stdev=122057.65, samples=119
iops : min= 2, max=16676, avg=2046.85, stdev=3814.30, samples=119
write: IOPS=868, BW=27.1MiB/s (28.5MB/s)(1629MiB/60010msec); 0 zone resets
clat (usec): min=11, max=387759, avg=531.82, stdev=4690.51
lat (usec): min=11, max=387760, avg=532.33, stdev=4690.53
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 44],
| 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 186], 60.00th=[ 433],
| 70.00th=[ 482], 80.00th=[ 502], 90.00th=[ 510], 95.00th=[ 515],
| 99.00th=[ 9372], 99.50th=[ 13698], 99.90th=[ 60031], 99.95th=[ 91751],
| 99.99th=[181404]
bw ( KiB/s): min= 64, max=227904, per=100.00%, avg=28748.66, stdev=52576.81, samples=116
iops : min= 2, max= 7122, avg=898.39, stdev=1643.03, samples=116
lat (usec) : 10=2.01%, 20=1.00%, 50=77.11%, 100=2.17%, 250=2.35%
lat (usec) : 500=7.73%, 750=6.04%, 1000=0.01%
lat (msec) : 2=0.55%, 4=0.01%, 10=0.13%, 20=0.72%, 50=0.07%
lat (msec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%
cpu : usr=0.82%, sys=12.38%, ctx=30577, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=121824,52127,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=63.4MiB/s (66.5MB/s), 63.4MiB/s-63.4MiB/s (66.5MB/s-66.5MB/s), io=3807MiB (3992MB), run=60010-60010msec
WRITE: bw=27.1MiB/s (28.5MB/s), 27.1MiB/s-27.1MiB/s (28.5MB/s-28.5MB/s), io=1629MiB (1708MB), run=60010-60010msec
root@bhyve01:~ #
Result:
- Read IOPS: 2,030
- Read Throughput: 66.5 MB/s
- Read Average Latency: ~0.261 ms
- Write IOPS: 868
- Write Throughput: 28.5 MB/s
- Write Average Latency: ~0.532 ms
Storage Performance test - 32 KB IO, 70% read / 30% write, 100% random, 2 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=70 \
> --numjobs=2 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=26.3MiB/s,w=11.4MiB/s][r=843,w=365 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=2): err= 0: pid=2303: Sun Aug 24 02:00:17 2025
read: IOPS=1200, BW=37.5MiB/s (39.3MB/s)(2256MiB/60128msec)
clat (usec): min=8, max=775248, avg=802.38, stdev=7245.08
lat (usec): min=8, max=775249, avg=802.59, stdev=7245.08
clat percentiles (usec):
| 1.00th=[ 38], 5.00th=[ 52], 10.00th=[ 330], 20.00th=[ 355],
| 30.00th=[ 371], 40.00th=[ 396], 50.00th=[ 424], 60.00th=[ 457],
| 70.00th=[ 515], 80.00th=[ 545], 90.00th=[ 635], 95.00th=[ 1004],
| 99.00th=[ 3032], 99.50th=[ 13566], 99.90th=[ 81265], 99.95th=[125305],
| 99.99th=[358613]
bw ( KiB/s): min= 3008, max=64576, per=100.00%, avg=38525.45, stdev=8463.63, samples=238
iops : min= 94, max= 2018, avg=1203.90, stdev=264.48, samples=238
write: IOPS=518, BW=16.2MiB/s (17.0MB/s)(975MiB/60128msec); 0 zone resets
clat (usec): min=12, max=307734, avg=1983.53, stdev=10498.96
lat (usec): min=12, max=307735, avg=1984.05, stdev=10498.97
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 58], 10.00th=[ 338], 20.00th=[ 363],
| 30.00th=[ 379], 40.00th=[ 408], 50.00th=[ 437], 60.00th=[ 465],
| 70.00th=[ 529], 80.00th=[ 553], 90.00th=[ 979], 95.00th=[ 7570],
| 99.00th=[ 49546], 99.50th=[ 79168], 99.90th=[143655], 99.95th=[173016],
| 99.99th=[258999]
bw ( KiB/s): min= 704, max=27520, per=100.00%, avg=16657.61, stdev=3706.74, samples=238
iops : min= 22, max= 860, avg=520.53, stdev=115.83, samples=238
lat (usec) : 10=0.11%, 20=0.19%, 50=4.63%, 100=0.19%, 250=0.04%
lat (usec) : 500=60.54%, 750=25.07%, 1000=2.79%
lat (msec) : 2=3.88%, 4=0.11%, 10=1.16%, 20=0.52%, 50=0.36%
lat (msec) : 100=0.27%, 250=0.12%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=0.32%, sys=4.10%, ctx=99873, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=72183,31190,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=37.5MiB/s (39.3MB/s), 37.5MiB/s-37.5MiB/s (39.3MB/s-39.3MB/s), io=2256MiB (2365MB), run=60128-60128msec
WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s), io=975MiB (1022MB), run=60128-60128msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=70 \
> --numjobs=2 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=26.3MiB/s,w=11.4MiB/s][r=843,w=365 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=2): err= 0: pid=2303: Sun Aug 24 02:00:17 2025
read: IOPS=1200, BW=37.5MiB/s (39.3MB/s)(2256MiB/60128msec)
clat (usec): min=8, max=775248, avg=802.38, stdev=7245.08
lat (usec): min=8, max=775249, avg=802.59, stdev=7245.08
clat percentiles (usec):
| 1.00th=[ 38], 5.00th=[ 52], 10.00th=[ 330], 20.00th=[ 355],
| 30.00th=[ 371], 40.00th=[ 396], 50.00th=[ 424], 60.00th=[ 457],
| 70.00th=[ 515], 80.00th=[ 545], 90.00th=[ 635], 95.00th=[ 1004],
| 99.00th=[ 3032], 99.50th=[ 13566], 99.90th=[ 81265], 99.95th=[125305],
| 99.99th=[358613]
bw ( KiB/s): min= 3008, max=64576, per=100.00%, avg=38525.45, stdev=8463.63, samples=238
iops : min= 94, max= 2018, avg=1203.90, stdev=264.48, samples=238
write: IOPS=518, BW=16.2MiB/s (17.0MB/s)(975MiB/60128msec); 0 zone resets
clat (usec): min=12, max=307734, avg=1983.53, stdev=10498.96
lat (usec): min=12, max=307735, avg=1984.05, stdev=10498.97
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 58], 10.00th=[ 338], 20.00th=[ 363],
| 30.00th=[ 379], 40.00th=[ 408], 50.00th=[ 437], 60.00th=[ 465],
| 70.00th=[ 529], 80.00th=[ 553], 90.00th=[ 979], 95.00th=[ 7570],
| 99.00th=[ 49546], 99.50th=[ 79168], 99.90th=[143655], 99.95th=[173016],
| 99.99th=[258999]
bw ( KiB/s): min= 704, max=27520, per=100.00%, avg=16657.61, stdev=3706.74, samples=238
iops : min= 22, max= 860, avg=520.53, stdev=115.83, samples=238
lat (usec) : 10=0.11%, 20=0.19%, 50=4.63%, 100=0.19%, 250=0.04%
lat (usec) : 500=60.54%, 750=25.07%, 1000=2.79%
lat (msec) : 2=3.88%, 4=0.11%, 10=1.16%, 20=0.52%, 50=0.36%
lat (msec) : 100=0.27%, 250=0.12%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu : usr=0.32%, sys=4.10%, ctx=99873, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=72183,31190,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=37.5MiB/s (39.3MB/s), 37.5MiB/s-37.5MiB/s (39.3MB/s-39.3MB/s), io=2256MiB (2365MB), run=60128-60128msec
WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s), io=975MiB (1022MB), run=60128-60128msec
root@bhyve01:~ #
Result:
- Read IOPS: 1,200
- Read Throughput: 39.3 MB/s
- Read Average Latency: ~0.8 ms
- Write IOPS: 518
- Write Throughput: 17 MB/s
- Write Average Latency: ~1.98 ms
Storage Performance test - 32 KB IO, 70% read / 30% write, 100% random, 4 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 4 processes
Jobs: 2 (f=2): [F(2),_(2)][100.0%][r=16.5MiB/s,w=6598KiB/s][r=528,w=206 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=4): err= 0: pid=3201: Sun Aug 24 05:58:26 2025
read: IOPS=3866, BW=121MiB/s (127MB/s)(7395MiB/61203msec)
clat (usec): min=7, max=535930, avg=66.15, stdev=2653.93
lat (usec): min=7, max=535931, avg=66.32, stdev=2653.94
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 38],
| 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 48], 95.00th=[ 53],
| 99.00th=[ 96], 99.50th=[ 117], 99.90th=[ 523], 99.95th=[ 594],
| 99.99th=[115868]
bw ( KiB/s): min=63744, max=1449840, per=100.00%, avg=128417.18, stdev=41953.42, samples=446
iops : min= 1992, max=45306, avg=4013.02, stdev=1311.02, samples=446
write: IOPS=1664, BW=52.0MiB/s (54.6MB/s)(3184MiB/61203msec); 0 zone resets
clat (usec): min=13, max=1586.4k, avg=2193.66, stdev=16734.16
lat (usec): min=13, max=1586.4k, avg=2194.21, stdev=16734.16
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 791],
| 30.00th=[ 1549], 40.00th=[ 1778], 50.00th=[ 1958], 60.00th=[ 2057],
| 70.00th=[ 2147], 80.00th=[ 2311], 90.00th=[ 2606], 95.00th=[ 2868],
| 99.00th=[ 3163], 99.50th=[ 3228], 99.90th=[189793], 99.95th=[316670],
| 99.99th=[675283]
bw ( KiB/s): min=29952, max=640214, per=100.00%, avg=54894.43, stdev=18074.87, samples=457
iops : min= 936, max=20005, avg=1715.44, stdev=564.81, samples=457
lat (usec) : 10=0.63%, 20=0.36%, 50=64.11%, 100=8.76%, 250=0.81%
lat (usec) : 500=0.52%, 750=0.60%, 1000=0.67%
lat (msec) : 2=9.56%, 4=13.92%, 10=0.01%, 50=0.01%, 100=0.01%
lat (msec) : 250=0.03%, 500=0.02%, 750=0.01%, 2000=0.01%
cpu : usr=0.49%, sys=6.48%, ctx=87574, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=236639,101890,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=7395MiB (7754MB), run=61203-61203msec
WRITE: bw=52.0MiB/s (54.6MB/s), 52.0MiB/s-52.0MiB/s (54.6MB/s-54.6MB/s), io=3184MiB (3339MB), run=61203-61203msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 4 processes
Jobs: 2 (f=2): [F(2),_(2)][100.0%][r=16.5MiB/s,w=6598KiB/s][r=528,w=206 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=4): err= 0: pid=3201: Sun Aug 24 05:58:26 2025
read: IOPS=3866, BW=121MiB/s (127MB/s)(7395MiB/61203msec)
clat (usec): min=7, max=535930, avg=66.15, stdev=2653.93
lat (usec): min=7, max=535931, avg=66.32, stdev=2653.94
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 37],
| 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 38],
| 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 48], 95.00th=[ 53],
| 99.00th=[ 96], 99.50th=[ 117], 99.90th=[ 523], 99.95th=[ 594],
| 99.99th=[115868]
bw ( KiB/s): min=63744, max=1449840, per=100.00%, avg=128417.18, stdev=41953.42, samples=446
iops : min= 1992, max=45306, avg=4013.02, stdev=1311.02, samples=446
write: IOPS=1664, BW=52.0MiB/s (54.6MB/s)(3184MiB/61203msec); 0 zone resets
clat (usec): min=13, max=1586.4k, avg=2193.66, stdev=16734.16
lat (usec): min=13, max=1586.4k, avg=2194.21, stdev=16734.16
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 791],
| 30.00th=[ 1549], 40.00th=[ 1778], 50.00th=[ 1958], 60.00th=[ 2057],
| 70.00th=[ 2147], 80.00th=[ 2311], 90.00th=[ 2606], 95.00th=[ 2868],
| 99.00th=[ 3163], 99.50th=[ 3228], 99.90th=[189793], 99.95th=[316670],
| 99.99th=[675283]
bw ( KiB/s): min=29952, max=640214, per=100.00%, avg=54894.43, stdev=18074.87, samples=457
iops : min= 936, max=20005, avg=1715.44, stdev=564.81, samples=457
lat (usec) : 10=0.63%, 20=0.36%, 50=64.11%, 100=8.76%, 250=0.81%
lat (usec) : 500=0.52%, 750=0.60%, 1000=0.67%
lat (msec) : 2=9.56%, 4=13.92%, 10=0.01%, 50=0.01%, 100=0.01%
lat (msec) : 250=0.03%, 500=0.02%, 750=0.01%, 2000=0.01%
cpu : usr=0.49%, sys=6.48%, ctx=87574, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=236639,101890,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=7395MiB (7754MB), run=61203-61203msec
WRITE: bw=52.0MiB/s (54.6MB/s), 52.0MiB/s-52.0MiB/s (54.6MB/s-54.6MB/s), io=3184MiB (3339MB), run=61203-61203msec
root@bhyve01:~ #
Result:
- Read IOPS: 3,866
- Read Throughput: 127 MB/s
- Read Average Latency: ~0.066 ms
- Write IOPS: 1664
- Write Throughput: 54.6 MB/s
- Write Average Latency: ~2.193 ms
Storage Performance test - 32 KB IO, 70% read / 30% write, 100% random, 8 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=70 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=70 \
> --numjobs=8 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 8 processes
Jobs: 8 (f=8): [m(8)][100.0%][r=111MiB/s,w=49.3MiB/s][r=3553,w=1577 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=8): err= 0: pid=3230: Sun Aug 24 06:12:20 2025
read: IOPS=3416, BW=107MiB/s (112MB/s)(6471MiB/60610msec)
clat (usec): min=7, max=1560.8k, avg=1484.29, stdev=28348.37
lat (usec): min=7, max=1560.8k, avg=1484.51, stdev=28348.37
clat percentiles (usec):
| 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 38],
| 20.00th=[ 39], 30.00th=[ 42], 40.00th=[ 420],
| 50.00th=[ 545], 60.00th=[ 693], 70.00th=[ 873],
| 80.00th=[ 1156], 90.00th=[ 1614], 95.00th=[ 1876],
| 99.00th=[ 3523], 99.50th=[ 4113], 99.90th=[ 170918],
| 99.95th=[ 683672], 99.99th=[1518339]
bw ( KiB/s): min= 4917, max=795008, per=100.00%, avg=131435.75, stdev=23809.09, samples=723
iops : min= 153, max=24844, avg=4107.36, stdev=744.03, samples=723
write: IOPS=1459, BW=45.6MiB/s (47.8MB/s)(2764MiB/60610msec); 0 zone resets
clat (usec): min=12, max=1549.0k, avg=1955.11, stdev=32881.07
lat (usec): min=12, max=1549.0k, avg=1955.68, stdev=32881.07
clat percentiles (usec):
| 1.00th=[ 44], 5.00th=[ 45], 10.00th=[ 46],
| 20.00th=[ 163], 30.00th=[ 338], 40.00th=[ 433],
| 50.00th=[ 562], 60.00th=[ 701], 70.00th=[ 881],
| 80.00th=[ 1156], 90.00th=[ 1614], 95.00th=[ 1909],
| 99.00th=[ 3654], 99.50th=[ 4359], 99.90th=[ 434111],
| 99.95th=[ 809501], 99.99th=[1535116]
bw ( KiB/s): min= 1920, max=343616, per=100.00%, avg=56121.69, stdev=10205.05, samples=719
iops : min= 60, max=10738, avg=1753.80, stdev=318.91, samples=719
lat (usec) : 10=0.26%, 20=0.51%, 50=26.55%, 100=1.27%, 250=1.68%
lat (usec) : 500=14.61%, 750=18.39%, 1000=11.96%
lat (msec) : 2=21.22%, 4=2.93%, 10=0.35%, 20=0.08%, 50=0.01%
lat (msec) : 100=0.02%, 250=0.04%, 500=0.04%, 750=0.02%, 1000=0.02%
lat (msec) : 2000=0.03%
cpu : usr=0.26%, sys=2.25%, ctx=215064, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=207078,88463,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=6471MiB (6786MB), run=60610-60610msec
WRITE: bw=45.6MiB/s (47.8MB/s), 45.6MiB/s-45.6MiB/s (47.8MB/s-47.8MB/s), io=2764MiB (2899MB), run=60610-60610msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=70 \
> --numjobs=8 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 8 processes
Jobs: 8 (f=8): [m(8)][100.0%][r=111MiB/s,w=49.3MiB/s][r=3553,w=1577 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=8): err= 0: pid=3230: Sun Aug 24 06:12:20 2025
read: IOPS=3416, BW=107MiB/s (112MB/s)(6471MiB/60610msec)
clat (usec): min=7, max=1560.8k, avg=1484.29, stdev=28348.37
lat (usec): min=7, max=1560.8k, avg=1484.51, stdev=28348.37
clat percentiles (usec):
| 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 38],
| 20.00th=[ 39], 30.00th=[ 42], 40.00th=[ 420],
| 50.00th=[ 545], 60.00th=[ 693], 70.00th=[ 873],
| 80.00th=[ 1156], 90.00th=[ 1614], 95.00th=[ 1876],
| 99.00th=[ 3523], 99.50th=[ 4113], 99.90th=[ 170918],
| 99.95th=[ 683672], 99.99th=[1518339]
bw ( KiB/s): min= 4917, max=795008, per=100.00%, avg=131435.75, stdev=23809.09, samples=723
iops : min= 153, max=24844, avg=4107.36, stdev=744.03, samples=723
write: IOPS=1459, BW=45.6MiB/s (47.8MB/s)(2764MiB/60610msec); 0 zone resets
clat (usec): min=12, max=1549.0k, avg=1955.11, stdev=32881.07
lat (usec): min=12, max=1549.0k, avg=1955.68, stdev=32881.07
clat percentiles (usec):
| 1.00th=[ 44], 5.00th=[ 45], 10.00th=[ 46],
| 20.00th=[ 163], 30.00th=[ 338], 40.00th=[ 433],
| 50.00th=[ 562], 60.00th=[ 701], 70.00th=[ 881],
| 80.00th=[ 1156], 90.00th=[ 1614], 95.00th=[ 1909],
| 99.00th=[ 3654], 99.50th=[ 4359], 99.90th=[ 434111],
| 99.95th=[ 809501], 99.99th=[1535116]
bw ( KiB/s): min= 1920, max=343616, per=100.00%, avg=56121.69, stdev=10205.05, samples=719
iops : min= 60, max=10738, avg=1753.80, stdev=318.91, samples=719
lat (usec) : 10=0.26%, 20=0.51%, 50=26.55%, 100=1.27%, 250=1.68%
lat (usec) : 500=14.61%, 750=18.39%, 1000=11.96%
lat (msec) : 2=21.22%, 4=2.93%, 10=0.35%, 20=0.08%, 50=0.01%
lat (msec) : 100=0.02%, 250=0.04%, 500=0.04%, 750=0.02%, 1000=0.02%
lat (msec) : 2000=0.03%
cpu : usr=0.26%, sys=2.25%, ctx=215064, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=207078,88463,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=6471MiB (6786MB), run=60610-60610msec
WRITE: bw=45.6MiB/s (47.8MB/s), 45.6MiB/s-45.6MiB/s (47.8MB/s-47.8MB/s), io=2764MiB (2899MB), run=60610-60610msec
root@bhyve01:~ #
Result:
- Read IOPS: 3,416
- Read Throughput: 112 MB/s
- Read Average Latency: ~1.48 ms
- Write IOPS: 1,459
- Write Throughput: 47.8 MB/s
- Write Average Latency: ~1.95 ms
32 KB IO, 0% read / 100 % write
Storage Performance test - 32 KB IO, 0% read / 100% write, 100% random, 1 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=1 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=0 \
> --numjobs=1 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.40
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=2240KiB/s][w=70 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=1): err= 0: pid=3278: Sun Aug 24 06:31:54 2025
write: IOPS=473, BW=14.8MiB/s (15.5MB/s)(889MiB/60049msec); 0 zone resets
clat (usec): min=10, max=964417, avg=2107.76, stdev=15377.08
lat (usec): min=11, max=964417, avg=2108.24, stdev=15377.09
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 44],
| 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 61],
| 70.00th=[ 363], 80.00th=[ 457], 90.00th=[ 6915], 95.00th=[ 9241],
| 99.00th=[ 12911], 99.50th=[ 83362], 99.90th=[204473], 99.95th=[274727],
| 99.99th=[658506]
bw ( KiB/s): min= 64, max=585984, per=100.00%, avg=15423.79, stdev=74369.79, samples=118
iops : min= 2, max=18312, avg=481.99, stdev=2324.06, samples=118
lat (usec) : 20=1.60%, 50=49.75%, 100=11.00%, 250=2.23%, 500=16.78%
lat (usec) : 750=6.76%, 1000=0.23%
lat (msec) : 2=0.03%, 4=0.10%, 10=9.74%, 20=0.90%, 50=0.11%
lat (msec) : 100=0.38%, 250=0.31%, 500=0.06%, 750=0.01%, 1000=0.01%
cpu : usr=0.24%, sys=2.27%, ctx=10275, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,28439,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=14.8MiB/s (15.5MB/s), 14.8MiB/s-14.8MiB/s (15.5MB/s-15.5MB/s), io=889MiB (932MB), run=60049-60049msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=0 \
> --numjobs=1 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
fio-3.40
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=2240KiB/s][w=70 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=1): err= 0: pid=3278: Sun Aug 24 06:31:54 2025
write: IOPS=473, BW=14.8MiB/s (15.5MB/s)(889MiB/60049msec); 0 zone resets
clat (usec): min=10, max=964417, avg=2107.76, stdev=15377.08
lat (usec): min=11, max=964417, avg=2108.24, stdev=15377.09
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 44],
| 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 61],
| 70.00th=[ 363], 80.00th=[ 457], 90.00th=[ 6915], 95.00th=[ 9241],
| 99.00th=[ 12911], 99.50th=[ 83362], 99.90th=[204473], 99.95th=[274727],
| 99.99th=[658506]
bw ( KiB/s): min= 64, max=585984, per=100.00%, avg=15423.79, stdev=74369.79, samples=118
iops : min= 2, max=18312, avg=481.99, stdev=2324.06, samples=118
lat (usec) : 20=1.60%, 50=49.75%, 100=11.00%, 250=2.23%, 500=16.78%
lat (usec) : 750=6.76%, 1000=0.23%
lat (msec) : 2=0.03%, 4=0.10%, 10=9.74%, 20=0.90%, 50=0.11%
lat (msec) : 100=0.38%, 250=0.31%, 500=0.06%, 750=0.01%, 1000=0.01%
cpu : usr=0.24%, sys=2.27%, ctx=10275, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,28439,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=14.8MiB/s (15.5MB/s), 14.8MiB/s-14.8MiB/s (15.5MB/s-15.5MB/s), io=889MiB (932MB), run=60049-60049msec
root@bhyve01:~ #
Result:
- Read IOPS: 0
- Read Throughput: 0 MB/s
- Read Average Latency: N/A
- Write IOPS: 473
- Write Throughput: 15.5 MB/s
- Write Average Latency: ~2.1 ms
Storage Performance test - 32 KB IO, 0% read / 100% write, 100% random, 2 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=2 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=0 \
> --numjobs=2 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 2 processes
Jobs: 1 (f=1): [F(1),_(1)][100.0%][w=5152KiB/s][w=161 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=2): err= 0: pid=3314: Sun Aug 24 06:45:33 2025
write: IOPS=1736, BW=54.3MiB/s (56.9MB/s)(3306MiB/60899msec); 0 zone resets
clat (usec): min=10, max=729133, avg=1131.06, stdev=2985.27
lat (usec): min=10, max=729134, avg=1131.64, stdev=2985.28
clat percentiles (usec):
| 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 363],
| 30.00th=[ 1045], 40.00th=[ 1352], 50.00th=[ 1401], 60.00th=[ 1418],
| 70.00th=[ 1450], 80.00th=[ 1516], 90.00th=[ 1614], 95.00th=[ 1745],
| 99.00th=[ 1893], 99.50th=[ 1909], 99.90th=[ 1942], 99.95th=[ 1975],
| 99.99th=[ 3589]
bw ( KiB/s): min=28096, max=733952, per=100.00%, avg=56716.63, stdev=52712.68, samples=237
iops : min= 878, max=22936, avg=1772.41, stdev=1647.35, samples=237
lat (usec) : 20=0.53%, 50=14.41%, 100=1.85%, 250=1.60%, 500=5.63%
lat (usec) : 750=2.02%, 1000=3.79%
lat (msec) : 2=70.14%, 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%
lat (msec) : 500=0.01%, 750=0.01%
cpu : usr=0.44%, sys=4.66%, ctx=117417, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,105780,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=54.3MiB/s (56.9MB/s), 54.3MiB/s-54.3MiB/s (56.9MB/s-56.9MB/s), io=3306MiB (3466MB), run=60899-60899msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=0 \
> --numjobs=2 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 2 processes
Jobs: 1 (f=1): [F(1),_(1)][100.0%][w=5152KiB/s][w=161 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=2): err= 0: pid=3314: Sun Aug 24 06:45:33 2025
write: IOPS=1736, BW=54.3MiB/s (56.9MB/s)(3306MiB/60899msec); 0 zone resets
clat (usec): min=10, max=729133, avg=1131.06, stdev=2985.27
lat (usec): min=10, max=729134, avg=1131.64, stdev=2985.28
clat percentiles (usec):
| 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 363],
| 30.00th=[ 1045], 40.00th=[ 1352], 50.00th=[ 1401], 60.00th=[ 1418],
| 70.00th=[ 1450], 80.00th=[ 1516], 90.00th=[ 1614], 95.00th=[ 1745],
| 99.00th=[ 1893], 99.50th=[ 1909], 99.90th=[ 1942], 99.95th=[ 1975],
| 99.99th=[ 3589]
bw ( KiB/s): min=28096, max=733952, per=100.00%, avg=56716.63, stdev=52712.68, samples=237
iops : min= 878, max=22936, avg=1772.41, stdev=1647.35, samples=237
lat (usec) : 20=0.53%, 50=14.41%, 100=1.85%, 250=1.60%, 500=5.63%
lat (usec) : 750=2.02%, 1000=3.79%
lat (msec) : 2=70.14%, 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%
lat (msec) : 500=0.01%, 750=0.01%
cpu : usr=0.44%, sys=4.66%, ctx=117417, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,105780,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=54.3MiB/s (56.9MB/s), 54.3MiB/s-54.3MiB/s (56.9MB/s-56.9MB/s), io=3306MiB (3466MB), run=60899-60899msec
root@bhyve01:~ #
Result:
- Read IOPS: 0
- Read Throughput: 0 MB/s
- Read Average Latency: N/A
- Write IOPS: 1,736
- Write Throughput: 56.9 MB/s
- Write Average Latency: ~1.13 ms
Storage Performance test - 32 KB IO, 0% read / 100% write, 100% random, 4 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=4 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=0 \
> --numjobs=4 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][w=41.9MiB/s][w=1342 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=4): err= 0: pid=3322: Sun Aug 24 06:56:00 2025
write: IOPS=1796, BW=56.2MiB/s (58.9MB/s)(3408MiB/60690msec); 0 zone resets
clat (usec): min=13, max=136994, avg=2197.11, stdev=1183.18
lat (usec): min=14, max=136995, avg=2197.72, stdev=1183.19
clat percentiles (usec):
| 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 1336],
| 30.00th=[ 1991], 40.00th=[ 2147], 50.00th=[ 2376], 60.00th=[ 2606],
| 70.00th=[ 2933], 80.00th=[ 3195], 90.00th=[ 3458], 95.00th=[ 3589],
| 99.00th=[ 3785], 99.50th=[ 3818], 99.90th=[ 3884], 99.95th=[ 3949],
| 99.99th=[ 4047]
bw ( KiB/s): min=33792, max=1122880, per=100.00%, avg=58297.55, stdev=25138.06, samples=476
iops : min= 1056, max=35090, avg=1821.80, stdev=785.56, samples=476
lat (usec) : 20=0.08%, 50=1.59%, 100=11.02%, 250=1.96%, 500=1.11%
lat (usec) : 750=1.11%, 1000=1.03%
lat (msec) : 2=13.74%, 4=68.35%, 10=0.03%, 250=0.01%
cpu : usr=0.28%, sys=2.71%, ctx=93879, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,109052,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=56.2MiB/s (58.9MB/s), 56.2MiB/s-56.2MiB/s (58.9MB/s-58.9MB/s), io=3408MiB (3573MB), run=60690-60690msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
> --directory=/STORAGE-DATA/bhyve-datastore \
> --ioengine=psync \
> --bs=32k \
> --size=100G \
> --rw=randrw \
> --rwmixread=0 \
> --numjobs=4 \
> --end_fsync=1 \
> --direct=1 \
> --runtime=60 \
> --group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][w=41.9MiB/s][w=1342 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=4): err= 0: pid=3322: Sun Aug 24 06:56:00 2025
write: IOPS=1796, BW=56.2MiB/s (58.9MB/s)(3408MiB/60690msec); 0 zone resets
clat (usec): min=13, max=136994, avg=2197.11, stdev=1183.18
lat (usec): min=14, max=136995, avg=2197.72, stdev=1183.19
clat percentiles (usec):
| 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 1336],
| 30.00th=[ 1991], 40.00th=[ 2147], 50.00th=[ 2376], 60.00th=[ 2606],
| 70.00th=[ 2933], 80.00th=[ 3195], 90.00th=[ 3458], 95.00th=[ 3589],
| 99.00th=[ 3785], 99.50th=[ 3818], 99.90th=[ 3884], 99.95th=[ 3949],
| 99.99th=[ 4047]
bw ( KiB/s): min=33792, max=1122880, per=100.00%, avg=58297.55, stdev=25138.06, samples=476
iops : min= 1056, max=35090, avg=1821.80, stdev=785.56, samples=476
lat (usec) : 20=0.08%, 50=1.59%, 100=11.02%, 250=1.96%, 500=1.11%
lat (usec) : 750=1.11%, 1000=1.03%
lat (msec) : 2=13.74%, 4=68.35%, 10=0.03%, 250=0.01%
cpu : usr=0.28%, sys=2.71%, ctx=93879, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,109052,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=56.2MiB/s (58.9MB/s), 56.2MiB/s-56.2MiB/s (58.9MB/s-58.9MB/s), io=3408MiB (3573MB), run=60690-60690msec
root@bhyve01:~ #
Result:
- Read IOPS: 0
- Read Throughput: 0 MB/s
- Read Average Latency: N/A
- Write IOPS: 1,796
- Write Throughput: 58.9 MB/s
- Write Average Latency: ~2.197 ms
Storage Performance test - 32 KB IO, 0% read / 100% write, 100% random, 8 job
Test Command:
fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
Output:
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 8 processes
Jobs: 8 (f=8): [w(8)][100.0%][w=31.1MiB/s][w=994 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=8): err= 0: pid=3380: Sun Aug 24 07:16:04 2025
write: IOPS=1508, BW=47.2MiB/s (49.4MB/s)(2867MiB/60800msec); 0 zone resets
clat (usec): min=14, max=1547.4k, avg=5236.22, stdev=8722.80
lat (usec): min=15, max=1547.4k, avg=5236.84, stdev=8722.80
clat percentiles (usec):
| 1.00th=[ 60], 5.00th=[ 66], 10.00th=[ 103], 20.00th=[ 2868],
| 30.00th=[ 5211], 40.00th=[ 5735], 50.00th=[ 5932], 60.00th=[ 6259],
| 70.00th=[ 6849], 80.00th=[ 7242], 90.00th=[ 7635], 95.00th=[ 7832],
| 99.00th=[ 8094], 99.50th=[ 8225], 99.90th=[ 8586], 99.95th=[16057],
| 99.99th=[62129]
bw ( KiB/s): min=24576, max=1010804, per=100.00%, avg=49232.93, stdev=11561.18, samples=949
iops : min= 768, max=31587, avg=1538.52, stdev=361.28, samples=949
lat (usec) : 20=0.01%, 50=0.05%, 100=9.42%, 250=5.20%, 500=0.20%
lat (usec) : 750=0.60%, 1000=0.57%
lat (msec) : 2=2.11%, 4=4.60%, 10=77.18%, 20=0.03%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%
cpu : usr=0.10%, sys=1.39%, ctx=82482, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,91739,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=47.2MiB/s (49.4MB/s), 47.2MiB/s-47.2MiB/s (49.4MB/s-49.4MB/s), io=2867MiB (3006MB), run=60800-60800msec
root@bhyve01:~ #
root@bhyve01:~ # fio --name=fio-test \
--directory=/STORAGE-DATA/bhyve-datastore \
--ioengine=psync \
--bs=32k \
--size=100G \
--rw=randrw \
--rwmixread=0 \
--numjobs=8 \
--end_fsync=1 \
--direct=1 \
--runtime=60 \
--group_reporting
fio-test: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1
...
fio-3.40
Starting 8 processes
Jobs: 8 (f=8): [w(8)][100.0%][w=31.1MiB/s][w=994 IOPS][eta 00m:00s]
fio-test: (groupid=0, jobs=8): err= 0: pid=3380: Sun Aug 24 07:16:04 2025
write: IOPS=1508, BW=47.2MiB/s (49.4MB/s)(2867MiB/60800msec); 0 zone resets
clat (usec): min=14, max=1547.4k, avg=5236.22, stdev=8722.80
lat (usec): min=15, max=1547.4k, avg=5236.84, stdev=8722.80
clat percentiles (usec):
| 1.00th=[ 60], 5.00th=[ 66], 10.00th=[ 103], 20.00th=[ 2868],
| 30.00th=[ 5211], 40.00th=[ 5735], 50.00th=[ 5932], 60.00th=[ 6259],
| 70.00th=[ 6849], 80.00th=[ 7242], 90.00th=[ 7635], 95.00th=[ 7832],
| 99.00th=[ 8094], 99.50th=[ 8225], 99.90th=[ 8586], 99.95th=[16057],
| 99.99th=[62129]
bw ( KiB/s): min=24576, max=1010804, per=100.00%, avg=49232.93, stdev=11561.18, samples=949
iops : min= 768, max=31587, avg=1538.52, stdev=361.28, samples=949
lat (usec) : 20=0.01%, 50=0.05%, 100=9.42%, 250=5.20%, 500=0.20%
lat (usec) : 750=0.60%, 1000=0.57%
lat (msec) : 2=2.11%, 4=4.60%, 10=77.18%, 20=0.03%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%
cpu : usr=0.10%, sys=1.39%, ctx=82482, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,91739,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=47.2MiB/s (49.4MB/s), 47.2MiB/s-47.2MiB/s (49.4MB/s-49.4MB/s), io=2867MiB (3006MB), run=60800-60800msec
root@bhyve01:~ #
Result:
- Read IOPS: 0
- Read Throughput: 0 MB/s
- Read Average Latency: N/A
- Write IOPS: 1,508
- Write Throughput: 49.4 MB/s
- Write Average Latency: ~5.24 ms
No comments:
Post a Comment