From 8f414e31c62bf7ea4a5197168c5852264dc1d397 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Fri, 26 Dec 2025 08:53:10 +0200 Subject: Update content for html --- ...5-04-05-f3s-kubernetes-with-freebsd-part-4.html | 116 +++++++++++++++++++- gemfeed/atom.xml | 120 ++++++++++++++++++++- 2 files changed, 232 insertions(+), 4 deletions(-) (limited to 'gemfeed') diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html index 35a60ed0..b680efac 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html @@ -13,7 +13,7 @@

f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs



-Published at 2025-04-04T23:21:01+03:00
+Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025

This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.

@@ -54,6 +54,13 @@
  • ⇢ ⇢ FreeBSD host ubench benchmark
  • ⇢ ⇢ FreeBSD VM @ Bhyve ubench benchmark
  • ⇢ ⇢ Rocky Linux VM @ Bhyve ubench benchmark
    • +
  • ⇢ Update: Improving Disk I/O Performance for etcd
    • +
  • ⇢ ⇢ The Problem
    • +
  • ⇢ ⇢ The Solution: Switch to NVMe Emulation
    • +
  • ⇢ ⇢ Step 1: Prepare the Guest OS
    • +
  • ⇢ ⇢ Step 2: Update the Bhyve Configuration
    • +
  • ⇢ ⇢ Benchmark Results
    • +
  • ⇢ ⇢ Important Notes
  • ⇢ Conclusion

    • Introduction


    @@ -587,6 +594,113 @@ Apr 4 23: Unfortunately, I wasn't able to find ubench in any of the Rocky Linux repositories. So, I skipped this test.

    +

    Update: Improving Disk I/O Performance for etcd


    +
    +Updated: Fri 26 Dec 08:51:23 EET 2025
    +
    +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default virtio-blk disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.
    +
    +

    The Problem


    +
    +The k3s logs showed etcd struggling with disk I/O:
    +
    +
    +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+
    +
    +A simple sync write benchmark confirmed the issue:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 31.7058 s, 258 kB/s
+
    +
    +

    The Solution: Switch to NVMe Emulation


    +
    +Bhyve's NVMe emulation provides significantly better I/O performance than virtio-blk.
    +
    +

    Step 1: Prepare the Guest OS


    +
    +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):
    +
    + +
    [root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF
+add_drivers+=" nvme nvme_core "
+hostonly=no
+EOF
+
+[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf
+[root@r0 ~]# dracut -f
+[root@r0 ~]# shutdown -h now
+
    +
    +The hostonly=no setting ensures the initramfs includes drivers for hardware not currently present. The use_devicesfile = 0 tells LVM to scan all block devices rather than only those recorded in /etc/lvm/devices/system.devices - this is important because the device path changes from /dev/vda to /dev/nvme0n1.
    +
    +

    Step 2: Update the Bhyve Configuration


    +
    +On the FreeBSD host, update the VM configuration to use NVMe:
    +
    + +
    paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+
    +
    +Change disk0_type from virtio-blk to nvme:
    +
    +
    +disk0_type="nvme"
+
    +
    +Then start the VM:
    +
    + +
    paul@f0:~ % doas vm start rocky
+
    +
    +

    Benchmark Results


    +
    +After switching to NVMe emulation, the sync write performance improved dramatically:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 0.330718 s, 24.8 MB/s
+
    +
    +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).
    +
    +The etcd metrics also showed healthy fsync latencies:
    +
    +
    +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+
    +
    +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.
    +
    +

    Important Notes


    +
    +

    Conclusion



    Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.
    diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index ad53209f..994c89f9 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ - 2025-12-26T01:27:25+02:00 + 2025-12-26T08:51:40+02:00 foo.zone feed To be in the .zone! @@ -9601,7 +9601,7 @@ __ejm\___/________dwb`---`______________________ f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs https://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html - 2025-04-04T23:21:01+03:00 + 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025 Paul Buetow aka snonux paul@dev.buetow.org @@ -9611,7 +9611,7 @@ __ejm\___/________dwb`---`______________________

    f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs



    -Published at 2025-04-04T23:21:01+03:00
    +Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025

    This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.

    @@ -9652,6 +9652,13 @@ __ejm\___/________dwb`---`______________________
  • ⇢ ⇢ FreeBSD host ubench benchmark
  • ⇢ ⇢ FreeBSD VM @ Bhyve ubench benchmark
  • ⇢ ⇢ Rocky Linux VM @ Bhyve ubench benchmark
    • +
  • ⇢ Update: Improving Disk I/O Performance for etcd
    • +
  • ⇢ ⇢ The Problem
    • +
  • ⇢ ⇢ The Solution: Switch to NVMe Emulation
    • +
  • ⇢ ⇢ Step 1: Prepare the Guest OS
    • +
  • ⇢ ⇢ Step 2: Update the Bhyve Configuration
    • +
  • ⇢ ⇢ Benchmark Results
    • +
  • ⇢ ⇢ Important Notes
  • ⇢ Conclusion

    • Introduction


    @@ -10185,6 +10192,113 @@ Apr 4 23: Unfortunately, I wasn't able to find ubench in any of the Rocky Linux repositories. So, I skipped this test.

    +

    Update: Improving Disk I/O Performance for etcd


    +
    +Updated: Fri 26 Dec 08:51:23 EET 2025
    +
    +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default virtio-blk disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.
    +
    +

    The Problem


    +
    +The k3s logs showed etcd struggling with disk I/O:
    +
    +
    +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+
    +
    +A simple sync write benchmark confirmed the issue:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 31.7058 s, 258 kB/s
+
    +
    +

    The Solution: Switch to NVMe Emulation


    +
    +Bhyve's NVMe emulation provides significantly better I/O performance than virtio-blk.
    +
    +

    Step 1: Prepare the Guest OS


    +
    +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):
    +
    + +
    [root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF
+add_drivers+=" nvme nvme_core "
+hostonly=no
+EOF
+
+[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf
+[root@r0 ~]# dracut -f
+[root@r0 ~]# shutdown -h now
+
    +
    +The hostonly=no setting ensures the initramfs includes drivers for hardware not currently present. The use_devicesfile = 0 tells LVM to scan all block devices rather than only those recorded in /etc/lvm/devices/system.devices - this is important because the device path changes from /dev/vda to /dev/nvme0n1.
    +
    +

    Step 2: Update the Bhyve Configuration


    +
    +On the FreeBSD host, update the VM configuration to use NVMe:
    +
    + +
    paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+
    +
    +Change disk0_type from virtio-blk to nvme:
    +
    +
    +disk0_type="nvme"
+
    +
    +Then start the VM:
    +
    + +
    paul@f0:~ % doas vm start rocky
+
    +
    +

    Benchmark Results


    +
    +After switching to NVMe emulation, the sync write performance improved dramatically:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 0.330718 s, 24.8 MB/s
+
    +
    +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).
    +
    +The etcd metrics also showed healthy fsync latencies:
    +
    +
    +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+
    +
    +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.
    +
    +

    Important Notes


    +
    +
      +
    • Do NOT use disk0_opts="nocache,direct" with NVMe emulation - in my testing this actually made performance worse.
      • +
    • The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot.
      • +
    • LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.
      • +

    Conclusion



    Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.
    -- cgit v1.2.3