From 8f414e31c62bf7ea4a5197168c5852264dc1d397 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Fri, 26 Dec 2025 08:53:10 +0200 Subject: Update content for html --- about/resources.html | 206 ++++++++++----------- ...5-04-05-f3s-kubernetes-with-freebsd-part-4.html | 116 +++++++++++- gemfeed/atom.xml | 120 +++++++++++- index.html | 2 +- uptime-stats.html | 2 +- 5 files changed, 337 insertions(+), 109 deletions(-) diff --git a/about/resources.html b/about/resources.html index 554d80ee..f622b0df 100644 --- a/about/resources.html +++ b/about/resources.html @@ -50,112 +50,112 @@ In random order:


Technical references



I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order:


Self-development and soft-skills books



In random order:


Here are notes of mine for some of the books

@@ -164,31 +164,31 @@ Some of these were in-person with exams; others were online learning lectures only. In random order:


Technical guides



These are not whole books, but guides (smaller or larger) which I found very useful. in random order:


Podcasts



@@ -197,61 +197,61 @@ In random order:


Podcasts I liked



I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests.


Newsletters I like



This is a mix of tech and non-tech newsletters I am subscribed to. In random order:


Magazines I like(d)



This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order:


Formal education



diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html index 35a60ed0..b680efac 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html @@ -13,7 +13,7 @@

f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs



-Published at 2025-04-04T23:21:01+03:00
+Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025

This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.

@@ -54,6 +54,13 @@
  • ⇢ ⇢ FreeBSD host ubench benchmark
  • ⇢ ⇢ FreeBSD VM @ Bhyve ubench benchmark
  • ⇢ ⇢ Rocky Linux VM @ Bhyve ubench benchmark
    • +
  • ⇢ Update: Improving Disk I/O Performance for etcd
    • +
  • ⇢ ⇢ The Problem
    • +
  • ⇢ ⇢ The Solution: Switch to NVMe Emulation
    • +
  • ⇢ ⇢ Step 1: Prepare the Guest OS
    • +
  • ⇢ ⇢ Step 2: Update the Bhyve Configuration
    • +
  • ⇢ ⇢ Benchmark Results
    • +
  • ⇢ ⇢ Important Notes
  • ⇢ Conclusion

    • Introduction


    @@ -587,6 +594,113 @@ Apr 4 23: Unfortunately, I wasn't able to find ubench in any of the Rocky Linux repositories. So, I skipped this test.

    +

    Update: Improving Disk I/O Performance for etcd


    +
    +Updated: Fri 26 Dec 08:51:23 EET 2025
    +
    +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default virtio-blk disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.
    +
    +

    The Problem


    +
    +The k3s logs showed etcd struggling with disk I/O:
    +
    +
    +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+
    +
    +A simple sync write benchmark confirmed the issue:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 31.7058 s, 258 kB/s
+
    +
    +

    The Solution: Switch to NVMe Emulation


    +
    +Bhyve's NVMe emulation provides significantly better I/O performance than virtio-blk.
    +
    +

    Step 1: Prepare the Guest OS


    +
    +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):
    +
    + +
    [root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF
+add_drivers+=" nvme nvme_core "
+hostonly=no
+EOF
+
+[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf
+[root@r0 ~]# dracut -f
+[root@r0 ~]# shutdown -h now
+
    +
    +The hostonly=no setting ensures the initramfs includes drivers for hardware not currently present. The use_devicesfile = 0 tells LVM to scan all block devices rather than only those recorded in /etc/lvm/devices/system.devices - this is important because the device path changes from /dev/vda to /dev/nvme0n1.
    +
    +

    Step 2: Update the Bhyve Configuration


    +
    +On the FreeBSD host, update the VM configuration to use NVMe:
    +
    + +
    paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+
    +
    +Change disk0_type from virtio-blk to nvme:
    +
    +
    +disk0_type="nvme"
+
    +
    +Then start the VM:
    +
    + +
    paul@f0:~ % doas vm start rocky
+
    +
    +

    Benchmark Results


    +
    +After switching to NVMe emulation, the sync write performance improved dramatically:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 0.330718 s, 24.8 MB/s
+
    +
    +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).
    +
    +The etcd metrics also showed healthy fsync latencies:
    +
    +
    +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+
    +
    +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.
    +
    +

    Important Notes


    +
    +

    Conclusion



    Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.
    diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index ad53209f..994c89f9 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ - 2025-12-26T01:27:25+02:00 + 2025-12-26T08:51:40+02:00 foo.zone feed To be in the .zone! @@ -9601,7 +9601,7 @@ __ejm\___/________dwb`---`______________________ f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs https://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html - 2025-04-04T23:21:01+03:00 + 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025 Paul Buetow aka snonux paul@dev.buetow.org @@ -9611,7 +9611,7 @@ __ejm\___/________dwb`---`______________________

    f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs



    -Published at 2025-04-04T23:21:01+03:00
    +Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025

    This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.

    @@ -9652,6 +9652,13 @@ __ejm\___/________dwb`---`______________________
  • ⇢ ⇢ FreeBSD host ubench benchmark
  • ⇢ ⇢ FreeBSD VM @ Bhyve ubench benchmark
  • ⇢ ⇢ Rocky Linux VM @ Bhyve ubench benchmark
    • +
  • ⇢ Update: Improving Disk I/O Performance for etcd
    • +
  • ⇢ ⇢ The Problem
    • +
  • ⇢ ⇢ The Solution: Switch to NVMe Emulation
    • +
  • ⇢ ⇢ Step 1: Prepare the Guest OS
    • +
  • ⇢ ⇢ Step 2: Update the Bhyve Configuration
    • +
  • ⇢ ⇢ Benchmark Results
    • +
  • ⇢ ⇢ Important Notes
  • ⇢ Conclusion

    • Introduction


    @@ -10185,6 +10192,113 @@ Apr 4 23: Unfortunately, I wasn't able to find ubench in any of the Rocky Linux repositories. So, I skipped this test.

    +

    Update: Improving Disk I/O Performance for etcd


    +
    +Updated: Fri 26 Dec 08:51:23 EET 2025
    +
    +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default virtio-blk disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.
    +
    +

    The Problem


    +
    +The k3s logs showed etcd struggling with disk I/O:
    +
    +
    +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+
    +
    +A simple sync write benchmark confirmed the issue:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 31.7058 s, 258 kB/s
+
    +
    +

    The Solution: Switch to NVMe Emulation


    +
    +Bhyve's NVMe emulation provides significantly better I/O performance than virtio-blk.
    +
    +

    Step 1: Prepare the Guest OS


    +
    +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):
    +
    + +
    [root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF
+add_drivers+=" nvme nvme_core "
+hostonly=no
+EOF
+
+[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf
+[root@r0 ~]# dracut -f
+[root@r0 ~]# shutdown -h now
+
    +
    +The hostonly=no setting ensures the initramfs includes drivers for hardware not currently present. The use_devicesfile = 0 tells LVM to scan all block devices rather than only those recorded in /etc/lvm/devices/system.devices - this is important because the device path changes from /dev/vda to /dev/nvme0n1.
    +
    +

    Step 2: Update the Bhyve Configuration


    +
    +On the FreeBSD host, update the VM configuration to use NVMe:
    +
    + +
    paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+
    +
    +Change disk0_type from virtio-blk to nvme:
    +
    +
    +disk0_type="nvme"
+
    +
    +Then start the VM:
    +
    + +
    paul@f0:~ % doas vm start rocky
+
    +
    +

    Benchmark Results


    +
    +After switching to NVMe emulation, the sync write performance improved dramatically:
    +
    + +
    [root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 0.330718 s, 24.8 MB/s
+
    +
    +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).
    +
    +The etcd metrics also showed healthy fsync latencies:
    +
    +
    +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+
    +
    +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.
    +
    +

    Important Notes


    +
    +
      +
    • Do NOT use disk0_opts="nocache,direct" with NVMe emulation - in my testing this actually made performance worse.
      • +
    • The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot.
      • +
    • LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.
      • +

    Conclusion



    Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.
    diff --git a/index.html b/index.html index 1265b929..c87ad5fc 100644 --- a/index.html +++ b/index.html @@ -13,7 +13,7 @@

    Hello!



    -This site was generated at 2025-12-26T01:27:25+02:00 by Gemtexter
    +This site was generated at 2025-12-26T08:51:39+02:00 by Gemtexter

    Welcome to the foo.zone!

    diff --git a/uptime-stats.html b/uptime-stats.html index fbad8eab..e6307ca5 100644 --- a/uptime-stats.html +++ b/uptime-stats.html @@ -13,7 +13,7 @@

    My machine uptime stats



    -This site was last updated at 2025-12-26T01:27:25+02:00
    +This site was last updated at 2025-12-26T08:51:39+02:00

    The following stats were collected via uptimed on all of my personal computers over many years and the output was generated by guprecords, the global uptime records stats analyser of mine.

    -- cgit v1.2.3