From f36be84efedd50f685f4069eec861ee11c8244f4 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Fri, 26 Dec 2025 08:53:13 +0200 Subject: Update content for gemtext --- ...25-04-05-f3s-kubernetes-with-freebsd-part-4.gmi | 100 ++++++++++++++++- ...4-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl | 93 +++++++++++++++- gemfeed/atom.xml | 120 ++++++++++++++++++++- 3 files changed, 308 insertions(+), 5 deletions(-) (limited to 'gemfeed') diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi index c093386f..512657e3 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi @@ -1,6 +1,6 @@ # f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs -> Published at 2025-04-04T23:21:01+03:00 +> Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025 This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines. @@ -40,6 +40,13 @@ This is the fourth blog post about the f3s series for self-hosting demands in a * ⇢ ⇢ ⇢ FreeBSD host `ubench` benchmark * ⇢ ⇢ ⇢ FreeBSD VM @ Bhyve `ubench` benchmark * ⇢ ⇢ ⇢ Rocky Linux VM @ Bhyve `ubench` benchmark +* ⇢ ⇢ Update: Improving Disk I/O Performance for etcd +* ⇢ ⇢ ⇢ The Problem +* ⇢ ⇢ ⇢ The Solution: Switch to NVMe Emulation +* ⇢ ⇢ ⇢ Step 1: Prepare the Guest OS +* ⇢ ⇢ ⇢ Step 2: Update the Bhyve Configuration +* ⇢ ⇢ ⇢ Benchmark Results +* ⇢ ⇢ ⇢ Important Notes * ⇢ ⇢ Conclusion ## Introduction @@ -498,6 +505,97 @@ Overall, Bhyve has a small overhead, but the CPU performance difference is negli Unfortunately, I wasn't able to find `ubench` in any of the Rocky Linux repositories. So, I skipped this test. +## Update: Improving Disk I/O Performance for etcd + +> Updated: Fri 26 Dec 08:51:23 EET 2025 + +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default `virtio-blk` disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation. + +### The Problem + +The k3s logs showed etcd struggling with disk I/O: + +``` +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"} +{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"} +``` + +A simple sync write benchmark confirmed the issue: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 31.7058 s, 258 kB/s +``` + +### The Solution: Switch to NVMe Emulation + +Bhyve's NVMe emulation provides significantly better I/O performance than `virtio-blk`. + +### Step 1: Prepare the Guest OS + +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation): + +```sh +[root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF +add_drivers+=" nvme nvme_core " +hostonly=no +EOF + +[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf +[root@r0 ~]# dracut -f +[root@r0 ~]# shutdown -h now +``` + +The `hostonly=no` setting ensures the initramfs includes drivers for hardware not currently present. The `use_devicesfile = 0` tells LVM to scan all block devices rather than only those recorded in `/etc/lvm/devices/system.devices` - this is important because the device path changes from `/dev/vda` to `/dev/nvme0n1`. + +### Step 2: Update the Bhyve Configuration + +On the FreeBSD host, update the VM configuration to use NVMe: + +```sh +paul@f0:~ % doas vm stop rocky +paul@f0:~ % doas vm configure rocky +``` + +Change `disk0_type` from `virtio-blk` to `nvme`: + +``` +disk0_type="nvme" +``` + +Then start the VM: + +```sh +paul@f0:~ % doas vm start rocky +``` + +### Benchmark Results + +After switching to NVMe emulation, the sync write performance improved dramatically: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 0.330718 s, 24.8 MB/s +``` + +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s). + +The etcd metrics also showed healthy fsync latencies: + +``` +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408 +``` + +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections. + +### Important Notes + +* Do NOT use `disk0_opts="nocache,direct"` with NVMe emulation - in my testing this actually made performance worse. +* The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot. +* LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path. + ## Conclusion Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD. diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl index b8428906..0cad941c 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl @@ -1,6 +1,6 @@ # f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs -> Published at 2025-04-04T23:21:01+03:00 +> Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025 This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines. @@ -466,6 +466,97 @@ Overall, Bhyve has a small overhead, but the CPU performance difference is negli Unfortunately, I wasn't able to find `ubench` in any of the Rocky Linux repositories. So, I skipped this test. +## Update: Improving Disk I/O Performance for etcd + +> Updated: Fri 26 Dec 08:51:23 EET 2025 + +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default `virtio-blk` disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation. + +### The Problem + +The k3s logs showed etcd struggling with disk I/O: + +``` +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"} +{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"} +``` + +A simple sync write benchmark confirmed the issue: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 31.7058 s, 258 kB/s +``` + +### The Solution: Switch to NVMe Emulation + +Bhyve's NVMe emulation provides significantly better I/O performance than `virtio-blk`. + +### Step 1: Prepare the Guest OS + +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation): + +```sh +[root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF +add_drivers+=" nvme nvme_core " +hostonly=no +EOF + +[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf +[root@r0 ~]# dracut -f +[root@r0 ~]# shutdown -h now +``` + +The `hostonly=no` setting ensures the initramfs includes drivers for hardware not currently present. The `use_devicesfile = 0` tells LVM to scan all block devices rather than only those recorded in `/etc/lvm/devices/system.devices` - this is important because the device path changes from `/dev/vda` to `/dev/nvme0n1`. + +### Step 2: Update the Bhyve Configuration + +On the FreeBSD host, update the VM configuration to use NVMe: + +```sh +paul@f0:~ % doas vm stop rocky +paul@f0:~ % doas vm configure rocky +``` + +Change `disk0_type` from `virtio-blk` to `nvme`: + +``` +disk0_type="nvme" +``` + +Then start the VM: + +```sh +paul@f0:~ % doas vm start rocky +``` + +### Benchmark Results + +After switching to NVMe emulation, the sync write performance improved dramatically: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 0.330718 s, 24.8 MB/s +``` + +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s). + +The etcd metrics also showed healthy fsync latencies: + +``` +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408 +``` + +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections. + +### Important Notes + +* Do NOT use `disk0_opts="nocache,direct"` with NVMe emulation - in my testing this actually made performance worse. +* The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot. +* LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path. + ## Conclusion Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD. diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index d225500a..89bf6a50 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ - 2025-12-26T01:27:25+02:00 + 2025-12-26T08:51:40+02:00 foo.zone feed To be in the .zone! @@ -9601,7 +9601,7 @@ __ejm\___/________dwb`---`______________________ f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi - 2025-04-04T23:21:01+03:00 + 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025 Paul Buetow aka snonux paul@dev.buetow.org @@ -9611,7 +9611,7 @@ __ejm\___/________dwb`---`______________________

f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs



-Published at 2025-04-04T23:21:01+03:00
+Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025

This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.

@@ -9652,6 +9652,13 @@ __ejm\___/________dwb`---`______________________
  • ⇢ ⇢ FreeBSD host ubench benchmark
  • ⇢ ⇢ FreeBSD VM @ Bhyve ubench benchmark
  • ⇢ ⇢ Rocky Linux VM @ Bhyve ubench benchmark
    • +
  • ⇢ Update: Improving Disk I/O Performance for etcd
    • +
  • ⇢ ⇢ The Problem
    • +
  • ⇢ ⇢ The Solution: Switch to NVMe Emulation
    • +
  • ⇢ ⇢ Step 1: Prepare the Guest OS
    • +
  • ⇢ ⇢ Step 2: Update the Bhyve Configuration
    • +
  • ⇢ ⇢ Benchmark Results
    • +
  • ⇢ ⇢ Important Notes
  • ⇢ Conclusion

    • Introduction


    @@ -10185,6 +10192,113 @@ Apr 4 23: Unfortunately, I wasn't able to find ubench in any of the Rocky Linux repositories. So, I skipped this test.

    +

    Update: Improving Disk I/O Performance for etcd


    +
    +Updated: Fri 26 Dec 08:51:23 EET 2025
    +
    +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default virtio-blk disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.
    +
    +

    The Problem


    +
    +The k3s logs showed etcd struggling with disk I/O:
    +
    +
    +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+
    +
    +A simple sync write benchmark confirmed the issue:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 31.7058 s, 258 kB/s
+
    +
    +

    The Solution: Switch to NVMe Emulation


    +
    +Bhyve's NVMe emulation provides significantly better I/O performance than virtio-blk.
    +
    +

    Step 1: Prepare the Guest OS


    +
    +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):
    +
    + +
    [root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF
+add_drivers+=" nvme nvme_core "
+hostonly=no
+EOF
+
+[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf
+[root@r0 ~]# dracut -f
+[root@r0 ~]# shutdown -h now
+
    +
    +The hostonly=no setting ensures the initramfs includes drivers for hardware not currently present. The use_devicesfile = 0 tells LVM to scan all block devices rather than only those recorded in /etc/lvm/devices/system.devices - this is important because the device path changes from /dev/vda to /dev/nvme0n1.
    +
    +

    Step 2: Update the Bhyve Configuration


    +
    +On the FreeBSD host, update the VM configuration to use NVMe:
    +
    + +
    paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+
    +
    +Change disk0_type from virtio-blk to nvme:
    +
    +
    +disk0_type="nvme"
+
    +
    +Then start the VM:
    +
    + +
    paul@f0:~ % doas vm start rocky
+
    +
    +

    Benchmark Results


    +
    +After switching to NVMe emulation, the sync write performance improved dramatically:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 0.330718 s, 24.8 MB/s
+
    +
    +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).
    +
    +The etcd metrics also showed healthy fsync latencies:
    +
    +
    +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+
    +
    +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.
    +
    +

    Important Notes


    +
    +
      +
    • Do NOT use disk0_opts="nocache,direct" with NVMe emulation - in my testing this actually made performance worse.
      • +
    • The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot.
      • +
    • LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.
      • +

    Conclusion



    Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.
    -- cgit v1.2.3