ZFS Managability

In this blog post, we will focus on ZFS from a manageability perspective. We will cover following topics

• ZFS Storage Status Monitoring
• Physical Disk Monitoring
• ZFS Capacity Monitoring
• ZFS Performance Monitoring
• Periodic and E-mail notifications

ZFS Storage Performance in my homelab

In my homelab I have Dell PowerEdge R620 Server with FreeBSD 14.3 and ZFS 2.2.7. I want to use this server for BHyVe server virtualization and run Virtual Machines on top of BHyVe hypervisor. 

In virtualized environment, the typical average I/O size differs based on workload running in virtual machines. Different applications generate distinct I/O patterns. 

• Databases and transactional systems: These often produce a large number of small, random I/O requests (e.g., 4KB, 8KB, or 16KB). This is because they frequently read and write small chunks of data to update records, log transactions, and access indexes.
• Virtual Desktop Infrastructure (VDI): VDI workloads are notoriously random and write-heavy, with an average I/O size often falling in the 24KB to 32KB range.
• File servers and data backups: These workloads typically generate large, sequential I/O requests (e.g., 64KB, 128KB, 256KB, or larger) as they read or write large files in a continuous stream. 

When I look at a typical enterprise cloud datacenter, where the types of workloads are not under your control, I usually observe the average I/O size between 40 KB/s and 64 KB/s. That's the reason why I typically test 32 KB I/O size, however, if you know the specific type of workload you are interested, you should test application specific I/O size.

LACP on FreeBSD

LACP stands for Link Aggregation Control Protocol. It’s a network protocol used to combine multiple physical network links into a single logical link to increase bandwidth and provide redundancy. It’s part of the IEEE 802.3ad standard (now 802.1AX). 

Here’s a breakdown of what it does and why it’s useful:

• Increases Bandwidth
• By bundling multiple links (like two or more Ethernet cables) between switches or between a switch and a server, the total throughput can be higher than a single link.
• Provides Redundancy
•  If one physical link fails, traffic is automatically rerouted over the remaining links, so the connection stays up.
• Dynamic Configuration
• LACP allows devices to automatically detect and configure link aggregation groups, making it easier to manage than static link aggregation.
• Load Balancing
• Traffic can be distributed across the aggregated links based on rules like source/destination IP, MAC addresses, or TCP/UDP ports. 

Let's configure and test it in my homelab.

LLDP on FreeBSD

LLDP stands for Link Layer Discovery Protocol. It’s a vendor-neutral Layer 2 protocol (defined in IEEE 802.1AB) that allows network devices (switches, routers, servers, firewalls, access points, phones, etc.) to advertise information about themselves to directly connected devices and to learn information about their neighbors.

In this short blog post we will install, enable and test LLDP on FreeBSD.

Home LAN (Cat5e vs WiFi) network throughputs

While testing Wi-Fi quality and network throughput on FreeBSD 14.3 drivers, I realized that before running any benchmarks, it’s important to document my home LAN topology and the network capacity across its zones. It’s essential to understand how different network technologies work, including the gap between their theoretical throughput and the actual achievable performance. 

For example, a Wi-Fi 5 (802.11ac) connection might advertise speeds up to 1.3 Gbps, but real-world performance is typically much lower due to factors like signal interference, channel width, and protocol overhead. Similarly, a 1 Gbps Ethernet link theoretically provides 1,000 Mbps, but after accounting for TCP/IP overhead and other factors, the actual throughput is closer to 940 Mbps. Another significant factor impacting real-world throughput is the use of Wi-Fi Mesh with wireless backhaul. While mesh systems improve coverage, they often introduce additional latency and bandwidth reduction because each hop between nodes consumes part of the available wireless spectrum for backhaul traffic. This means that, in practice, a device connected to a secondary mesh node (Extender) might experience only half or even less of the primary link’s bandwidth. Knowing these differences helps set realistic expectations and troubleshoot performance issues effectively. 

A picture is worth a thousand words, so here is a diagram illustrating both the theoretical and real-world throughput values in my home network setup.

Home LAN zones and Network Throughput

 You can find all the details in the remainder of this blog post.

FreeBSD 14.3 - Laptop computer with Intel Wi-Fi 5 (802.11ac) Wireless-AC 9260

FreeBSD 14.3 should have significantly improved support of WiFi adapters. 

Let's test it in Dell Precision 5530 with Intel Wi-Fi 5 (802.11ac) Wireless-AC 9x6x [Thunder Peak].

IPv6 - Part 3 - FreeBSD IPv6 configuration for Vodafone (ex-UPC)

Vodafone is one of the internet providers I use in my home lab setup here in Czechia.

I have been told they can enable IPv6 in my modem/router on request and it is not enabled by default. Anyway, it took them few minutes to reconfigure my modem/router to support IPv6. After this reconfiguratio, I connected my FreeBSD machine to the network segment we use as point-to-point (P2P /30) for IPv4. For IPv6, there is /64 subnet, where I can connect my IPv6 device.

Logical Network schema is depicted below.

Logical Network Schema

Let's start with configuration.

Public DNS Servers (Resolvers)

While writing my blog post series about IPv6, I realized it would be useful to document publicly available DNS servers. 

In this blog post I will documente DNS IP addresses of DNS Servers from Google, Clouflare, Quad9, Cisco's OpenDNS.

There are weel know IPv4 DNS addresses like 8.8.8.8  and 8.8.4.4, but there are others. DNS Servers are nowadays very usefull for secuurity protection like Phishing Protection, Optional content filtering, etc.

And last but not least, do you know IPv6 addresses of those DNS services.

IPv6 - Part 4 - FreeBSD IPv6 configuration for Starnet

Starnet is one of the internet providers I use in my home lab setup here in Czechia.

I have been told they are IPv6 ready, so I connected my FreeBSD machine to the network segment we use as point-to-point (P2P /30) for IPv4. For IPv6, there is /64 subnet, where I can connect my IPv6 device.

Logical Network schema is depicted below.

Logical Network Schema

 

Let's start with configuration.

 

FreeBSD and Edimax N150 Wi-Fi USB network interface

I have found old Edimax N150 Wi-Fi USB network interface and would like to use it in FreeBSD 14.2. for some IoT project. I have not used Wi-Fi on FreeBSD for ages, so let's try it.

It is worth to mention that Wi-Fi network interface can be in three different modes

• Station (client) - ifconfig wlan0 mode sta
• Monitor - ifconfig wlan0 mode monitor
• Access Point - ifconfig wlan0 mode hostap

Access Point (ifconfig wlan0 mode hostap) is great in situations you would like to allow multiple Stations to connect, but the rtwn driver in FreeBSD does not support Access Point (hostap) mode.  

Monitor mode on a wireless interface (ifconfig wlan0 mode monitor) is a special mode used primarily for passive packet capturing and wireless debugging, not for normal network communication. This mode should be supported by rtwn driver in FreeBSD, but I did not tested.

Station/Client (sta) mode is supported and it is actually the only mode we will cover in this blog post.

Let's do a configuration, setup, and some performance tests ...

FreeBSD Update and Upgrade process

The Base FreeBSD System vs Third-Party Software

First of all, it is important to understand that FreeBSD hase The Base FreeBSD system and Third-Party Software.

The Base FreeBSD System is the core part of FreeBSD that includes the kernel, standard system utilities, libraries, configuration files, and essential tools required to run and manage the system. You manage it using Admin Tool freebsd-update. Tool freebsd-update is still widely used, but the FreeBSD project is gradually moving toward pkgbase tool where The Base FreeBSD System is splited into packages like FreeBSD-runtime, FreeBSD-lib, FreeBSD-kernel, etc. You will be able to manage the base system with pkg just like third-party software. It will be more modular and modern than freebsd-update, but pkgbase is not yet officially supported on RELEASE versions, therefore freebsd-update is still production ready tool for update and upgrade of The Base FreeBSD System.

Third-Party software in FreeBSD is any application or tool not included in the base system, such as web servers, editors, databases, programming languages, and desktop environments. You manage it using the pkg package manager or by Ports Collection (source code + make).

Picture is worth 1,000 words, so I have prepared visualization to understand the difference between The Base FreeBSD System and Third-Party Software.

The Base FreeBSD System and Third-Party Software

FreeBSD Router with DNS and DHCP Servers

I use FreeBSD routers in my home labs and here is my typical router configuration.

Router configuration is based on 

• General FreeBSD System Configuration
• DHCP Server Configuration
• DNS (BIND) Configuration

Wirequard VPN on FreeBSD

I use site-to-site VPNs between datacenter and two remote locations and I had some strange issues with OpenVPN site-to-site performance of one particular VPN link to remote location, but the same OpenVPN configuration worked perfectly fine in another remote location. It was probably related to some UDP magic of that particular ISP. Monthly cost of that residential link is $20, so there was unrealistic to open support ticket with ISP and do some deep troubleshooting. Instead of that, I tried WireGuard VPN and it worked like a charm.

 

That was the reason I switched to from OpenVPN to WireGuard VPN and here is configuration of WireGuard VPN Server with two VPN clients. 

I have FreeBSD based VPN box in each location and below is the diagram with WireGuard interfaces (wg0) in each datacenter. WireGuard in data center is obviously WireGuard Server (172.16.100.254/24) and in remote locations I have WireGuard Clients (172.16.100.1/24 and 172.16.100.2/24).

 

WireGuard site-to-site VPN Toplogy

HowTo switch to FreeBSD legacy console

In terms of FreeBSD console, there are two settings typically set in /boot/loader.conf to affect early boot behavior.

kern.vty=sc

This setting tells FreeBSD to use the "sc" (syscons) console driver instead of the newer "vt" (Newcons) driver.

• sc is the older legacy text console system.
• vt (the default in modern FreeBSD versions) supports Unicode, better font rendering, and KMS (Kernel Mode Setting) for modern graphics.

You might set kern.vty=sc for:

• Compatibility with older hardware
• Simpler framebuffer requirements
• Easier use in virtual machines or serial consoles

hw.vga.textmode=1 

This setting forces the VGA hardware to remain in text mode during the boot process and afterward. When used with kern.vty=sc, it helps to avoid switching to graphics mode. It is useful on real hardware where mode switching causes flicker, or to avoid issues with VMs or KVMs that don't like graphics mode. 

It ensures that the system boots and runs entirely in VGA 80x25 text mode, improving compatibility and avoiding graphical issues.

How to disable all hardware network offload features and Jumbo Frames in Debian and FreeBSD

FreeBSD commands to disable hardware network offload features

# Disable hardware offload features

ifconfig vmx0 -rxcsum -rxcsum6 -txcsum -txcsum6 -tso -lro -vlanhwtag -vlanhwtso -vlanhwcsum -mextpg

 

# Disable Jumbo Frames

ifconfig vmx0 mtu 1500

 

#Check current settings

ifconfig vmx0

Debian commands to disable hardware network offload features

#!/usr/bin/sh

 

#Disable Jumbo Frames

ip link set dev ens192 mtu 1500

 

#Disable Large Receive Offload (LRO)

ethtool -K ens192 lro off

#Disable TCP Segmentation Offload (TSO)

ethtool -K ens192 tso off

#Disable Generic Receive Offload (GRO)

ethtool -K ens192 gro off

#Disable Generic Segmentation Offload (GSO)

ethtool -K ens192 gso off

#Disable Scatter-Gather (SG)

ethtool -K ens192 sg off

#Disable RX Checksumming

ethtool -K ens192 rx-checksumming off

#Disable TX Checksumming

ethtool -K ens192 tx-checksumming off

#Disable RX VLAN offloading

ethtool -K ens192 rx-vlan-offload off 

#Disable TX VLAN offloading

ethtool -K ens192 tx-vlan-offload off 

#Disable TX UDP Tunnel Segmentation Offload

ethtool -K ens192 tx-udp_tnl-segmentation off

#Disable Transmit UDP Tunnel Checksum Segmentation Offload 

ethtool -K ens192 tx-udp_tnl-csum-segmentation off

#Check current settings

ethtool -k ens192

How to switch TCP Stack and Congestion Algorithms in FreeBSD

The TCP stack and congestion control algorithms are core components of any modern operating system's networking infrastructure. They directly influence the performance, reliability, and efficiency of data communication over networks, especially over the internet or WANs.

Role of the TCP Stack

The TCP (Transmission Control Protocol) stack is part of the OS kernel that

1. Manages Reliable Transport
• Handles packet ordering, retransmission, and acknowledgment (ACKs).
• Ensures no data is lost, duplicated, or delivered out of order.
2. Implements Flow Control
• Uses the sliding window mechanism to prevent overwhelming the receiver.
3. Implements Congestion Control
• Reacts to network conditions (e.g., packet loss or delay) to adjust transmission rates.
4. Integrates with the OS Networking Subsystem
• Interacts with the IP layer, NIC drivers, and user-space sockets (bind(), send(), etc.).
• Supports features like NAT traversal, QoS, TCP Fast Open, and ECN (Explicit Congestion Notification).

Role of Congestion Algorithms

Congestion Algorithms determine how fast TCP can send data, especially under varying network conditions. Modern algorithms:

• Adjust the congestion window (cwnd) dynamically.
• Try to avoid congestion (proactively) and recover quickly if it happens.

Let's deep dive into options we have in FreeBSD 14 and how we can use them ...

How to set package site in FreeBSD

When you want to explicitly set packahe site in FreeBSD use shell command below ...

setenv PACKAGESITE https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest

Then you can use package commands as usual. 

pkg update

pkg search

pkg install

etc.

Useful "troubleshooting" commands in FreeBSD

Let's document some useful, not only "troubleshooting", FreeBSD commands. 

I will split those commands into following categories:

• General OS Management
• RAM Management
• Network Management
• Storage Management
• USB device Management
• Power Management
• Display / Graphics Cards Management

IPv6 - Part 2 - IPv6 configuration in FreeBSD

Before configuration of IPv6 in FreeBSD, I highly recommend to read my (Part 1) blog post "Everything I need to know about IPv6 address blocks" to get familiar with IPv6 basic concepts.

In all three sites of my home lab environment I use FreeBSD as a primary Operating System. I'll start exploring IPv6 right on the FreeBSD operating system.

The IPv6 configuration in FreeBSD is usually easy. ISP router typically supports SLAAC, so you can dynamically get IPv6 addresses, IPv6 default route, and even IPv6 DNS addresses from ISP router. The second option how to get IPv6 configuration from ISP router is DHCPv6.

Let’s explore and configure both SLAAC and DHCPv6 in my environment, and document all the details in this blog post - Part 2 of my blog series on IPv6.

IPv6 - Part 1 - Everything I need to know about IPv6 addresses

IPv6 (Internet Protocol version 6) was officially released as a standard in December 1998, with the publication of RFC 2460 by the IETF (Internet Engineering Task Force). 

 

It was usable for interoperability testing between Unix-like systems and Windows-based systems since 2006, when Microsoft included native IPv6 support in Windows Vista.

 

In 2012, major ISPs and websites enabled IPv6 permanently. It is called World IPv6 Launch Day.

 

It’s now 2025, so I think it’s time to test IPv6 readiness across the three ISPs I use for my home lab networks here in Czechia, Central Europe.

 

These ISPs are

1. Vodafone (Global Telco Provider) - ISP for my apartement where is small home lab
2. StarNet (Czech Telco Provider) - ISP for my house where is large home lab
3. Cloud4com (Czech Cloud Service Provider) - ISP for my lab in data center (cloud-based)

My home lab network, shown below, has been running on IPv4 for nearly 20 years. Is it already the right time to switch to IPv6?

 

Logical Network Schema

The idea is to keep IPv4 network as is and create new IPv6 network in paralel to do a Proof of Concept and get more familiar with IPv6. I can afford it because all my sites are fully virtualized, therefore it is not a problem to spin up additional IPv6 routers or devices in any of three sites.

 

In this Part 1 blog post, I would like to cover everything I need to know about IPv6 addresses. In future blog posts, I'll cover configuration details and real experience with IPv6.

 

How to boot FreeBSD into Single User Mode

When something went wrong, it is good to boot into single user mode (without the user/root authorization) and do some maintenance tasks.

Boot in to a single user mode

First of all, you must have access to FreeBSD console to manage boot process, because you have to somehowe initiated reboot of the system. When you have access to console keyboard, simply press CTRL+ALT+DEL. Another option is hardware reset or power-ofF & power-on, but this is not a graceful reboot and you can damage something.

During the boot sequence, there is "Bestie boot menu" where you can simply select option 2 by pressing key 2.

Bestie boot menu

Change read-only filesystem to read-write

When FreeBSD is booted into a single user mode, the file system is in read-only mode for safety.

When you want to change someting in file system or even change root or user password, you have to remount file system from read-only mode into read-write mode. 

For UFS

Below is the sequence of commands to do so if you have UFS file system.

mount -u / 

mount -a

Command (mount -u /) remounts the root filesystem (/) using the options specified (or defaults from /etc/fstab), without unmounting it.

Command (mount -a) mounts the rest of the filesystems defined in /etc/fstab.

For ZFS

Below is the sequence of commands for ZFS file system.

zfs set readonly=off zroot/ROOT/default
zfs mount -a

 

Commands above are self explanatory.

Work in a single user mode

Now you can do a troubleshooting or fixing some problems in single user operating system, where nobody else can login into the system and noone will interfere with you.

Alternative to single user mode

You can boot your system from FreeBSD boot media (ISO, USB Stick, etc.) into a recovery mode. It is essentialy running system from Live CD/USB disk. In such mode you have to mount disk filesystems by yourself to have read/write access to it.