Update content for gemtext

6 files changed, 413 insertions, 110 deletions

diff --git a/about/resources.gmi b/about/resources.gmi
index bd5fed9d..b31bab05 100644
--- a/about/resources.gmi
+++ b/about/resources.gmi
@@ -35,110 +35,110 @@ You won't find any links on this site because, over time, the links will break.
 
 In random order:
 
-* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible
-* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly
-* Pro Puppet; James Turnbull, Jeffrey McCune; Apress
+* Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt 
 * Raku Fundamentals; Moritz Lenz; Apress
-* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly
-* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers 
-* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook
-* Data Science at the Command Line; Jeroen Janssens; O'Reilly
+* Site Reliability Engineering; How Google runs production systems; O'Reilly
+* C++ Programming Language; Bjarne Stroustrup;
 * Funktionale Programmierung; Peter Pepper; Springer
-* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton
-* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook
-* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications
-* Polished Ruby Programming; Jeremy Evans; Packt Publishing
-* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson
-* Leanring eBPF; Liz Rice; O'Reilly
-* The Docker Book; James Turnbull; Kindle
-* Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt 
-* Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook
-* Java ist auch eine Insel; Christian Ullenboom; 
-* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly
+* Perl New Features; Joshua McAdams, brian d foy; Perl School
+* Pro Puppet; James Turnbull, Jeffrey McCune; Apress
+* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly
 * DNS and BIND; Cricket Liu; O'Reilly
-* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly
-* Ultimate Go Notebook; Bill Kennedy
-* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press
+* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications
+* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly
+* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional
+* Leanring eBPF; Liz Rice; O'Reilly
 * The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress
-* Terraform Cookbook; Mikael Krief; Packt Publishing
-* Perl New Features; Joshua McAdams, brian d foy; Perl School
-* C++ Programming Language; Bjarne Stroustrup;
+* Concurrency in Go; Katherine Cox-Buday; O'Reilly
+* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press
+* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson
+* Higher Order Perl; Mark Dominus; Morgan Kaufmann
 * The Pragmatic Programmer; David Thomas; Addison-Wesley
 * Developing Games in Java; David Brackeen and others...; New Riders
-* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications
-* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible
+* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf
+* Ultimate Go Notebook; Bill Kennedy
+* Effective awk programming; Arnold Robbins; O'Reilly
+* Java ist auch eine Insel; Christian Ullenboom; 
+* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton
 * Modern Perl; Chromatic ; Onyx Neon Press
-* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly
+* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications
+* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly
+* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly
+* Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook
+* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly
 * Systemprogrammierung in Go; Frank Müller; dpunkt
-* Effective Java; Joshua Bloch; Addison-Wesley Professional
-* Effective awk programming; Arnold Robbins; O'Reilly
-* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press
-* Concurrency in Go; Katherine Cox-Buday; O'Reilly
-* Higher Order Perl; Mark Dominus; Morgan Kaufmann
-* Site Reliability Engineering; How Google runs production systems; O'Reilly
-* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional
 * Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers
+* The Docker Book; James Turnbull; Kindle
+* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers 
+* Terraform Cookbook; Mikael Krief; Packt Publishing
+* Polished Ruby Programming; Jeremy Evans; Packt Publishing
+* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible
+* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press
+* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible
+* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook
+* Effective Java; Joshua Bloch; Addison-Wesley Professional
 * Raku Recipes; J.J. Merelo; Apress
-* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf
-* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly
-* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly
 * Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner
+* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly
+* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly
+* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook
+* Data Science at the Command Line; Jeroen Janssens; O'Reilly
 
 ## Technical references
 
 I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order:
 
-* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly
-* Go: Design Patterns for Real-World Projects; Mat Ryer; Packt
+* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley
 * Implementing Service Level Objectives; Alex Hidalgo; O'Reilly
-* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly
 * Relayd and Httpd Mastery; Michael W Lucas
-* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley
+* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly
+* Go: Design Patterns for Real-World Projects; Mat Ryer; Packt
 * The Linux Programming Interface; Michael Kerrisk; No Starch Press 
-* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley
+* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley
+* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly
 
 ## Self-development and soft-skills books
 
 In random order:
 
-* Stop starting, start finishing; Arne Roock; Lean-Kanban University  
-* Atomic Habits; James Clear; Random House Business
-* The Joy of Missing Out; Christina Crook; New Society Publishers
-* Getting Things Done; David Allen
+* The Power of Now; Eckhard Tolle; Yellow Kite
 * Ultralearning; Anna Laurent; Self-published via Amazon
-* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion
-* Digital Minimalism; Cal Newport; Portofolio Penguin
-* The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook 
-* Slow Productivity; Cal Newport; Penguin Random House
 * Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook
-* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing
-* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook
-* Eat That Frog; Brian Tracy
-* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press
-* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK
-* Psycho-Cybernetics; Maxwell Maltz; Perigee Books
-* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business
-* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook
-* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook
-* Ultralearning; Scott Young; Thorsons
-* Influence without Authority; A. Cohen, D. Bradford; Wiley
-* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly
-* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne
+* Getting Things Done; David Allen
 * The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books
+* Meditation for Mortals, Oliver Burkeman, Audiobook
 * The Bullet Journal Method; Ryder Carroll; Fourth Estate
+* Ultralearning; Scott Young; Thorsons
+* Soft Skills; John Sommez; Manning Publications
 * The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd
-* The Power of Now; Eckhard Tolle; Yellow Kite
-* Deep Work; Cal Newport; Piatkus
+* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook
+* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME)
 * The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME)
+* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select
 * So Good They Can't Ignore You; Cal Newport; Business Plus
+* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook
+* 101 Essays that change the way you think; Brianna Wiest; Audiobook
 * The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook
-* Soft Skills; John Sommez; Manning Publications
+* Stop starting, start finishing; Arne Roock; Lean-Kanban University  
+* Atomic Habits; James Clear; Random House Business
+* Digital Minimalism; Cal Newport; Portofolio Penguin
+* The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook 
 * Eat That Frog!; Brian Tracy; Hodder Paperbacks
-* Meditation for Mortals, Oliver Burkeman, Audiobook
-* 101 Essays that change the way you think; Brianna Wiest; Audiobook
+* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook
+* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business
+* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly
+* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK
+* Eat That Frog; Brian Tracy
+* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne
 * The Good Enough Job; Simone Stolzoff; Ebury Edge
-* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select
-* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME)
+* The Joy of Missing Out; Christina Crook; New Society Publishers
+* Slow Productivity; Cal Newport; Penguin Random House
+* Deep Work; Cal Newport; Piatkus
+* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press
+* Influence without Authority; A. Cohen, D. Bradford; Wiley
+* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion
+* Psycho-Cybernetics; Maxwell Maltz; Perigee Books
+* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing
 
 => ../notes/index.gmi Here are notes of mine for some of the books
 
@@ -146,30 +146,30 @@ In random order:
 
 Some of these were in-person with exams; others were online learning lectures only. In random order:
 
+* Ultimate Go Programming; Bill Kennedy; O'Reilly Online
 * Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online
-* MySQL Deep Dive Workshop; 2-day on-site training
+* Functional programming lecture; Remote University of Hagen
+* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need)
+* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon
+* Protocol buffers; O'Reilly Online
+* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training
+* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online
 * The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online
+* Structure and Interpretation of Computer Programs; Harold Abelson and more...; 
+* MySQL Deep Dive Workshop; 2-day on-site training
 * Developing IaC with Terraform (with Live Lessons); O'Reilly Online
-* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon
 * AWS Immersion Day; Amazon; 1-day interactive online training 
-* Protocol buffers; O'Reilly Online
-* Functional programming lecture; Remote University of Hagen
-* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. 
 * Scripting Vim; Damian Conway; O'Reilly Online
-* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need)
-* Structure and Interpretation of Computer Programs; Harold Abelson and more...; 
 * Apache Tomcat Best Practises; 3-day on-site training
-* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training
-* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online
-* Ultimate Go Programming; Bill Kennedy; O'Reilly Online
+* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. 
 
 ## Technical guides
 
 These are not whole books, but guides (smaller or larger) which I found very useful. in random order:
 
-* How CPUs work at https://cpu.land
-* Advanced Bash-Scripting Guide 
 * Raku Guide at https://raku.guide  
+* Advanced Bash-Scripting Guide 
+* How CPUs work at https://cpu.land
 
 ## Podcasts
 
@@ -177,58 +177,58 @@ These are not whole books, but guides (smaller or larger) which I found very use
 
 In random order:
 
+* Hidden Brain
+* BSD Now [BSD]
+* Backend Banter
 * Deep Questions with Cal Newport
+* Maintainable
+* The ProdCast (Google SRE Podcast)
 * Dev Interrupted
 * The Pragmatic Engineer Podcast
-* Hidden Brain
-* The Changelog Podcast(s)
-* The ProdCast (Google SRE Podcast)
-* Wednesday Wisdom
-* Fork Around And Find Out
 * Modern Mentor
-* Maintainable
-* Pratical AI
-* Cup o' Go [Golang]
+* Fork Around And Find Out
+* Wednesday Wisdom
 * Fallthrough [Golang]
-* BSD Now [BSD]
-* Backend Banter
+* The Changelog Podcast(s)
+* Cup o' Go [Golang]
+* Pratical AI
 
 ### Podcasts I liked
 
 I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests.
 
-* Java Pub House
-* CRE: Chaosradio Express [german]
 * FLOSS weekly
+* CRE: Chaosradio Express [german]
+* Ship It (predecessor of Fork Around And Find Out)
 * Go Time (predecessor of fallthrough)
 * Modern Mentor
-* Ship It (predecessor of Fork Around And Find Out)
+* Java Pub House
 
 ## Newsletters I like
 
 This is a mix of tech and non-tech newsletters I am subscribed to. In random order:
 
-* The Imperfectionist
-* Register Spill
-* Applied Go Weekly Newsletter
+* Changelog News
 * The Valuable Dev
 * byteSizeGo
+* VK Newsletter
 * Ruby Weekly
+* Monospace Mentor
 * Golang Weekly
 * The Pragmatic Engineer
-* VK Newsletter
-* Changelog News
 * Andreas Brandhorst Newsletter (Sci-Fi author)
-* Monospace Mentor
+* Register Spill
+* Applied Go Weekly Newsletter
+* The Imperfectionist
 
 ## Magazines I like(d)
 
 This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order:
 
-* freeX (not published anymore)
-* Linux Magazine
 * Linux User
 * LWN (online only)
+* freeX (not published anymore)
+* Linux Magazine
 
 # Formal education
 
diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi
index c093386f..512657e3 100644
--- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi
+++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi
@@ -1,6 +1,6 @@
 # f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
 
-> Published at 2025-04-04T23:21:01+03:00
+> Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025
 
 This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.
 
@@ -40,6 +40,13 @@ This is the fourth blog post about the f3s series for self-hosting demands in a
 * ⇢ ⇢ ⇢ FreeBSD host `ubench` benchmark
 * ⇢ ⇢ ⇢ FreeBSD VM @ Bhyve `ubench` benchmark
 * ⇢ ⇢ ⇢ Rocky Linux VM @ Bhyve `ubench` benchmark
+* ⇢ ⇢ Update: Improving Disk I/O Performance for etcd
+* ⇢ ⇢ ⇢ The Problem
+* ⇢ ⇢ ⇢ The Solution: Switch to NVMe Emulation
+* ⇢ ⇢ ⇢ Step 1: Prepare the Guest OS
+* ⇢ ⇢ ⇢ Step 2: Update the Bhyve Configuration
+* ⇢ ⇢ ⇢ Benchmark Results
+* ⇢ ⇢ ⇢ Important Notes
 * ⇢ ⇢ Conclusion
 
 ## Introduction
@@ -498,6 +505,97 @@ Overall, Bhyve has a small overhead, but the CPU performance difference is negli
 
 Unfortunately, I wasn't able to find `ubench` in any of the Rocky Linux repositories. So, I skipped this test.
 
+## Update: Improving Disk I/O Performance for etcd
+
+> Updated: Fri 26 Dec 08:51:23 EET 2025
+
+After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default `virtio-blk` disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.
+
+### The Problem
+
+The k3s logs showed etcd struggling with disk I/O:
+
+```
+{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+```
+
+A simple sync write benchmark confirmed the issue:
+
+```sh
+[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 31.7058 s, 258 kB/s
+```
+
+### The Solution: Switch to NVMe Emulation
+
+Bhyve's NVMe emulation provides significantly better I/O performance than `virtio-blk`.
+
+### Step 1: Prepare the Guest OS
+
+Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):
+
+```sh
+[root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF
+add_drivers+=" nvme nvme_core "
+hostonly=no
+EOF
+
+[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf
+[root@r0 ~]# dracut -f
+[root@r0 ~]# shutdown -h now
+```
+
+The `hostonly=no` setting ensures the initramfs includes drivers for hardware not currently present. The `use_devicesfile = 0` tells LVM to scan all block devices rather than only those recorded in `/etc/lvm/devices/system.devices` - this is important because the device path changes from `/dev/vda` to `/dev/nvme0n1`.
+
+### Step 2: Update the Bhyve Configuration
+
+On the FreeBSD host, update the VM configuration to use NVMe:
+
+```sh
+paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+```
+
+Change `disk0_type` from `virtio-blk` to `nvme`:
+
+```
+disk0_type="nvme"
+```
+
+Then start the VM:
+
+```sh
+paul@f0:~ % doas vm start rocky
+```
+
+### Benchmark Results
+
+After switching to NVMe emulation, the sync write performance improved dramatically:
+
+```sh
+[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 0.330718 s, 24.8 MB/s
+```
+
+That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).
+
+The etcd metrics also showed healthy fsync latencies:
+
+```
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+```
+
+Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.
+
+### Important Notes
+
+* Do NOT use `disk0_opts="nocache,direct"` with NVMe emulation - in my testing this actually made performance worse.
+* The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot.
+* LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.
+
 ## Conclusion
 
 Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.
diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl
index b8428906..0cad941c 100644
--- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl
+++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl
@@ -1,6 +1,6 @@
 # f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
 
-> Published at 2025-04-04T23:21:01+03:00
+> Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025
 
 This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.
 
@@ -466,6 +466,97 @@ Overall, Bhyve has a small overhead, but the CPU performance difference is negli
 
 Unfortunately, I wasn't able to find `ubench` in any of the Rocky Linux repositories. So, I skipped this test.
 
+## Update: Improving Disk I/O Performance for etcd
+
+> Updated: Fri 26 Dec 08:51:23 EET 2025
+
+After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default `virtio-blk` disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.
+
+### The Problem
+
+The k3s logs showed etcd struggling with disk I/O:
+
+```
+{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+```
+
+A simple sync write benchmark confirmed the issue:
+
+```sh
+[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 31.7058 s, 258 kB/s
+```
+
+### The Solution: Switch to NVMe Emulation
+
+Bhyve's NVMe emulation provides significantly better I/O performance than `virtio-blk`.
+
+### Step 1: Prepare the Guest OS
+
+Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):
+
+```sh
+[root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF
+add_drivers+=" nvme nvme_core "
+hostonly=no
+EOF
+
+[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf
+[root@r0 ~]# dracut -f
+[root@r0 ~]# shutdown -h now
+```
+
+The `hostonly=no` setting ensures the initramfs includes drivers for hardware not currently present. The `use_devicesfile = 0` tells LVM to scan all block devices rather than only those recorded in `/etc/lvm/devices/system.devices` - this is important because the device path changes from `/dev/vda` to `/dev/nvme0n1`.
+
+### Step 2: Update the Bhyve Configuration
+
+On the FreeBSD host, update the VM configuration to use NVMe:
+
+```sh
+paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+```
+
+Change `disk0_type` from `virtio-blk` to `nvme`:
+
+```
+disk0_type="nvme"
+```
+
+Then start the VM:
+
+```sh
+paul@f0:~ % doas vm start rocky
+```
+
+### Benchmark Results
+
+After switching to NVMe emulation, the sync write performance improved dramatically:
+
+```sh
+[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync
+8192000 bytes copied, 0.330718 s, 24.8 MB/s
+```
+
+That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).
+
+The etcd metrics also showed healthy fsync latencies:
+
+```
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+```
+
+Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.
+
+### Important Notes
+
+* Do NOT use `disk0_opts="nocache,direct"` with NVMe emulation - in my testing this actually made performance worse.
+* The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot.
+* LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.
+
 ## Conclusion
 
 Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.
diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml
index d225500a..89bf6a50 100644
--- a/gemfeed/atom.xml
+++ b/gemfeed/atom.xml
@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <feed xmlns="http://www.w3.org/2005/Atom">
-    <updated>2025-12-26T01:27:25+02:00</updated>
+    <updated>2025-12-26T08:51:40+02:00</updated>
     <title>foo.zone feed</title>
     <subtitle>To be in the .zone!</subtitle>
     <link href="gemini://foo.zone/gemfeed/atom.xml" rel="self" />
@@ -9601,7 +9601,7 @@ __ejm\___/________dwb`---`______________________
         <title>f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</title>
         <link href="gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi" />
         <id>gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi</id>
-        <updated>2025-04-04T23:21:01+03:00</updated>
+        <updated>2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025</updated>
         <author>
             <name>Paul Buetow aka snonux</name>
             <email>paul@dev.buetow.org</email>
@@ -9611,7 +9611,7 @@ __ejm\___/________dwb`---`______________________
             <div xmlns="http://www.w3.org/1999/xhtml">
                 <h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-4-rocky-linux-bhyve-vms'>f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</h1><br />
 <br />
-<span class='quote'>Published at 2025-04-04T23:21:01+03:00</span><br />
+<span class='quote'>Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025</span><br />
 <br />
 <span>This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.</span><br />
 <br />
@@ -9652,6 +9652,13 @@ __ejm\___/________dwb`---`______________________
 <li>⇢ ⇢ <a href='#freebsd-host-ubench-benchmark'>FreeBSD host <span class='inlinecode'>ubench</span> benchmark</a></li>
 <li>⇢ ⇢ <a href='#freebsd-vm--bhyve-ubench-benchmark'>FreeBSD VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</a></li>
 <li>⇢ ⇢ <a href='#rocky-linux-vm--bhyve-ubench-benchmark'>Rocky Linux VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</a></li>
+<li>⇢ <a href='#update-improving-disk-io-performance-for-etcd'>Update: Improving Disk I/O Performance for etcd</a></li>
+<li>⇢ ⇢ <a href='#the-problem'>The Problem</a></li>
+<li>⇢ ⇢ <a href='#the-solution-switch-to-nvme-emulation'>The Solution: Switch to NVMe Emulation</a></li>
+<li>⇢ ⇢ <a href='#step-1-prepare-the-guest-os'>Step 1: Prepare the Guest OS</a></li>
+<li>⇢ ⇢ <a href='#step-2-update-the-bhyve-configuration'>Step 2: Update the Bhyve Configuration</a></li>
+<li>⇢ ⇢ <a href='#benchmark-results'>Benchmark Results</a></li>
+<li>⇢ ⇢ <a href='#important-notes'>Important Notes</a></li>
 <li>⇢ <a href='#conclusion'>Conclusion</a></li>
 </ul><br />
 <h2 style='display: inline' id='introduction'>Introduction</h2><br />
@@ -10185,6 +10192,113 @@ Apr  <font color="#000000">4</font> <font color="#000000">23</font>:<font color=
 <br />
 <span>Unfortunately, I wasn&#39;t able to find <span class='inlinecode'>ubench</span> in any of the Rocky Linux repositories. So, I skipped this test.</span><br />
 <br />
+<h2 style='display: inline' id='update-improving-disk-io-performance-for-etcd'>Update: Improving Disk I/O Performance for etcd</h2><br />
+<br />
+<span class='quote'>Updated: Fri 26 Dec 08:51:23 EET 2025</span><br />
+<br />
+<span>After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd&#39;s sync writes were extremely slow - around 250 kB/s with the default <span class='inlinecode'>virtio-blk</span> disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.</span><br />
+<br />
+<h3 style='display: inline' id='the-problem'>The Problem</h3><br />
+<br />
+<span>The k3s logs showed etcd struggling with disk I/O:</span><br />
+<br />
+<pre>
+{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"}
+{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"}
+</pre>
+<br />
+<span>A simple sync write benchmark confirmed the issue:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>[root@r0 ~]<i><font color="silver"># dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync</font></i>
+<font color="#000000">8192000</font> bytes copied, <font color="#000000">31.7058</font> s, <font color="#000000">258</font> kB/s
+</pre>
+<br />
+<h3 style='display: inline' id='the-solution-switch-to-nvme-emulation'>The Solution: Switch to NVMe Emulation</h3><br />
+<br />
+<span>Bhyve&#39;s NVMe emulation provides significantly better I/O performance than <span class='inlinecode'>virtio-blk</span>.</span><br />
+<br />
+<h3 style='display: inline' id='step-1-prepare-the-guest-os'>Step 1: Prepare the Guest OS</h3><br />
+<br />
+<span>Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>[root@r0 ~]<i><font color="silver"># cat &gt; /etc/dracut.conf.d/nvme.conf &lt;&lt; EOF</font></i>
+add_drivers+=<font color="#808080">" nvme nvme_core "</font>
+hostonly=no
+EOF
+
+[root@r0 ~]<i><font color="silver"># sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf</font></i>
+[root@r0 ~]<i><font color="silver"># dracut -f</font></i>
+[root@r0 ~]<i><font color="silver"># shutdown -h now</font></i>
+</pre>
+<br />
+<span>The <span class='inlinecode'>hostonly=no</span> setting ensures the initramfs includes drivers for hardware not currently present. The <span class='inlinecode'>use_devicesfile = 0</span> tells LVM to scan all block devices rather than only those recorded in <span class='inlinecode'>/etc/lvm/devices/system.devices</span> - this is important because the device path changes from <span class='inlinecode'>/dev/vda</span> to <span class='inlinecode'>/dev/nvme0n1</span>.</span><br />
+<br />
+<h3 style='display: inline' id='step-2-update-the-bhyve-configuration'>Step 2: Update the Bhyve Configuration</h3><br />
+<br />
+<span>On the FreeBSD host, update the VM configuration to use NVMe:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas vm stop rocky
+paul@f0:~ % doas vm configure rocky
+</pre>
+<br />
+<span>Change <span class='inlinecode'>disk0_type</span> from <span class='inlinecode'>virtio-blk</span> to <span class='inlinecode'>nvme</span>:</span><br />
+<br />
+<pre>
+disk0_type="nvme"
+</pre>
+<br />
+<span>Then start the VM:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>paul@f0:~ % doas vm start rocky
+</pre>
+<br />
+<h3 style='display: inline' id='benchmark-results'>Benchmark Results</h3><br />
+<br />
+<span>After switching to NVMe emulation, the sync write performance improved dramatically:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>[root@r0 ~]<i><font color="silver"># dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync</font></i>
+<font color="#000000">8192000</font> bytes copied, <font color="#000000">0.330718</font> s, <font color="#000000">24.8</font> MB/s
+</pre>
+<br />
+<span>That&#39;s approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).</span><br />
+<br />
+<span>The etcd metrics also showed healthy fsync latencies:</span><br />
+<br />
+<pre>
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396
+etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408
+</pre>
+<br />
+<span>Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.</span><br />
+<br />
+<h3 style='display: inline' id='important-notes'>Important Notes</h3><br />
+<br />
+<ul>
+<li>Do NOT use <span class='inlinecode'>disk0_opts="nocache,direct"</span> with NVMe emulation - in my testing this actually made performance worse.</li>
+<li>The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won&#39;t boot.</li>
+<li>LVM&#39;s devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.</li>
+</ul><br />
 <h2 style='display: inline' id='conclusion'>Conclusion</h2><br />
 <br />
 <span>Having Linux VMs running inside FreeBSD&#39;s Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.</span><br />
diff --git a/index.gmi b/index.gmi
index cca11f1e..948f226f 100644
--- a/index.gmi
+++ b/index.gmi
@@ -1,6 +1,6 @@
 # Hello!
 
-> This site was generated at 2025-12-26T01:27:25+02:00 by `Gemtexter`
+> This site was generated at 2025-12-26T08:51:39+02:00 by `Gemtexter`
 
 Welcome to the foo.zone!
 
diff --git a/uptime-stats.gmi b/uptime-stats.gmi
index b9efb0e7..d18f38dd 100644
--- a/uptime-stats.gmi
+++ b/uptime-stats.gmi
@@ -1,6 +1,6 @@
 # My machine uptime stats
 
-> This site was last updated at 2025-12-26T01:27:25+02:00
+> This site was last updated at 2025-12-26T08:51:39+02:00
 
 The following stats were collected via `uptimed` on all of my personal computers over many years and the output was generated by `guprecords`, the global uptime records stats analyser of mine.