diff options
| -rw-r--r-- | about/resources.gmi | 206 | ||||
| -rw-r--r-- | gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi | 100 | ||||
| -rw-r--r-- | gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl | 93 | ||||
| -rw-r--r-- | gemfeed/atom.xml | 120 | ||||
| -rw-r--r-- | index.gmi | 2 | ||||
| -rw-r--r-- | uptime-stats.gmi | 2 |
6 files changed, 413 insertions, 110 deletions
diff --git a/about/resources.gmi b/about/resources.gmi index bd5fed9d..b31bab05 100644 --- a/about/resources.gmi +++ b/about/resources.gmi @@ -35,110 +35,110 @@ You won't find any links on this site because, over time, the links will break. In random order: -* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible -* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly -* Pro Puppet; James Turnbull, Jeffrey McCune; Apress +* Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt * Raku Fundamentals; Moritz Lenz; Apress -* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly -* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers -* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook -* Data Science at the Command Line; Jeroen Janssens; O'Reilly +* Site Reliability Engineering; How Google runs production systems; O'Reilly +* C++ Programming Language; Bjarne Stroustrup; * Funktionale Programmierung; Peter Pepper; Springer -* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton -* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook -* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications -* Polished Ruby Programming; Jeremy Evans; Packt Publishing -* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson -* Leanring eBPF; Liz Rice; O'Reilly -* The Docker Book; James Turnbull; Kindle -* Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt -* Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook -* Java ist auch eine Insel; Christian Ullenboom; -* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly +* Perl New Features; Joshua McAdams, brian d foy; Perl School +* Pro Puppet; James Turnbull, Jeffrey McCune; Apress +* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly * DNS and BIND; Cricket Liu; O'Reilly -* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly -* Ultimate Go Notebook; Bill Kennedy -* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press +* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications +* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly +* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional +* Leanring eBPF; Liz Rice; O'Reilly * The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress -* Terraform Cookbook; Mikael Krief; Packt Publishing -* Perl New Features; Joshua McAdams, brian d foy; Perl School -* C++ Programming Language; Bjarne Stroustrup; +* Concurrency in Go; Katherine Cox-Buday; O'Reilly +* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press +* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson +* Higher Order Perl; Mark Dominus; Morgan Kaufmann * The Pragmatic Programmer; David Thomas; Addison-Wesley * Developing Games in Java; David Brackeen and others...; New Riders -* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications -* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf +* Ultimate Go Notebook; Bill Kennedy +* Effective awk programming; Arnold Robbins; O'Reilly +* Java ist auch eine Insel; Christian Ullenboom; +* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton * Modern Perl; Chromatic ; Onyx Neon Press -* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly +* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications +* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly +* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly +* Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook +* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly * Systemprogrammierung in Go; Frank Müller; dpunkt -* Effective Java; Joshua Bloch; Addison-Wesley Professional -* Effective awk programming; Arnold Robbins; O'Reilly -* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press -* Concurrency in Go; Katherine Cox-Buday; O'Reilly -* Higher Order Perl; Mark Dominus; Morgan Kaufmann -* Site Reliability Engineering; How Google runs production systems; O'Reilly -* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional * Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers +* The Docker Book; James Turnbull; Kindle +* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* Terraform Cookbook; Mikael Krief; Packt Publishing +* Polished Ruby Programming; Jeremy Evans; Packt Publishing +* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible +* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press +* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook +* Effective Java; Joshua Bloch; Addison-Wesley Professional * Raku Recipes; J.J. Merelo; Apress -* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf -* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly -* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly * Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner +* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly +* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly +* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook +* Data Science at the Command Line; Jeroen Janssens; O'Reilly ## Technical references I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order: -* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly -* Go: Design Patterns for Real-World Projects; Mat Ryer; Packt +* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley * Implementing Service Level Objectives; Alex Hidalgo; O'Reilly -* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly * Relayd and Httpd Mastery; Michael W Lucas -* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley +* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly +* Go: Design Patterns for Real-World Projects; Mat Ryer; Packt * The Linux Programming Interface; Michael Kerrisk; No Starch Press -* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley +* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley +* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly ## Self-development and soft-skills books In random order: -* Stop starting, start finishing; Arne Roock; Lean-Kanban University -* Atomic Habits; James Clear; Random House Business -* The Joy of Missing Out; Christina Crook; New Society Publishers -* Getting Things Done; David Allen +* The Power of Now; Eckhard Tolle; Yellow Kite * Ultralearning; Anna Laurent; Self-published via Amazon -* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion -* Digital Minimalism; Cal Newport; Portofolio Penguin -* The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook -* Slow Productivity; Cal Newport; Penguin Random House * Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook -* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing -* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook -* Eat That Frog; Brian Tracy -* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press -* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK -* Psycho-Cybernetics; Maxwell Maltz; Perigee Books -* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business -* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook -* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook -* Ultralearning; Scott Young; Thorsons -* Influence without Authority; A. Cohen, D. Bradford; Wiley -* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly -* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne +* Getting Things Done; David Allen * The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books +* Meditation for Mortals, Oliver Burkeman, Audiobook * The Bullet Journal Method; Ryder Carroll; Fourth Estate +* Ultralearning; Scott Young; Thorsons +* Soft Skills; John Sommez; Manning Publications * The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd -* The Power of Now; Eckhard Tolle; Yellow Kite -* Deep Work; Cal Newport; Piatkus +* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook +* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME) * The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME) +* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select * So Good They Can't Ignore You; Cal Newport; Business Plus +* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook +* 101 Essays that change the way you think; Brianna Wiest; Audiobook * The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook -* Soft Skills; John Sommez; Manning Publications +* Stop starting, start finishing; Arne Roock; Lean-Kanban University +* Atomic Habits; James Clear; Random House Business +* Digital Minimalism; Cal Newport; Portofolio Penguin +* The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook * Eat That Frog!; Brian Tracy; Hodder Paperbacks -* Meditation for Mortals, Oliver Burkeman, Audiobook -* 101 Essays that change the way you think; Brianna Wiest; Audiobook +* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook +* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business +* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly +* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK +* Eat That Frog; Brian Tracy +* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne * The Good Enough Job; Simone Stolzoff; Ebury Edge -* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select -* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME) +* The Joy of Missing Out; Christina Crook; New Society Publishers +* Slow Productivity; Cal Newport; Penguin Random House +* Deep Work; Cal Newport; Piatkus +* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press +* Influence without Authority; A. Cohen, D. Bradford; Wiley +* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion +* Psycho-Cybernetics; Maxwell Maltz; Perigee Books +* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing => ../notes/index.gmi Here are notes of mine for some of the books @@ -146,30 +146,30 @@ In random order: Some of these were in-person with exams; others were online learning lectures only. In random order: +* Ultimate Go Programming; Bill Kennedy; O'Reilly Online * Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online -* MySQL Deep Dive Workshop; 2-day on-site training +* Functional programming lecture; Remote University of Hagen +* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) +* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon +* Protocol buffers; O'Reilly Online +* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training +* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online * The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online +* Structure and Interpretation of Computer Programs; Harold Abelson and more...; +* MySQL Deep Dive Workshop; 2-day on-site training * Developing IaC with Terraform (with Live Lessons); O'Reilly Online -* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon * AWS Immersion Day; Amazon; 1-day interactive online training -* Protocol buffers; O'Reilly Online -* Functional programming lecture; Remote University of Hagen -* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. * Scripting Vim; Damian Conway; O'Reilly Online -* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) -* Structure and Interpretation of Computer Programs; Harold Abelson and more...; * Apache Tomcat Best Practises; 3-day on-site training -* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training -* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online -* Ultimate Go Programming; Bill Kennedy; O'Reilly Online +* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. ## Technical guides These are not whole books, but guides (smaller or larger) which I found very useful. in random order: -* How CPUs work at https://cpu.land -* Advanced Bash-Scripting Guide * Raku Guide at https://raku.guide +* Advanced Bash-Scripting Guide +* How CPUs work at https://cpu.land ## Podcasts @@ -177,58 +177,58 @@ These are not whole books, but guides (smaller or larger) which I found very use In random order: +* Hidden Brain +* BSD Now [BSD] +* Backend Banter * Deep Questions with Cal Newport +* Maintainable +* The ProdCast (Google SRE Podcast) * Dev Interrupted * The Pragmatic Engineer Podcast -* Hidden Brain -* The Changelog Podcast(s) -* The ProdCast (Google SRE Podcast) -* Wednesday Wisdom -* Fork Around And Find Out * Modern Mentor -* Maintainable -* Pratical AI -* Cup o' Go [Golang] +* Fork Around And Find Out +* Wednesday Wisdom * Fallthrough [Golang] -* BSD Now [BSD] -* Backend Banter +* The Changelog Podcast(s) +* Cup o' Go [Golang] +* Pratical AI ### Podcasts I liked I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests. -* Java Pub House -* CRE: Chaosradio Express [german] * FLOSS weekly +* CRE: Chaosradio Express [german] +* Ship It (predecessor of Fork Around And Find Out) * Go Time (predecessor of fallthrough) * Modern Mentor -* Ship It (predecessor of Fork Around And Find Out) +* Java Pub House ## Newsletters I like This is a mix of tech and non-tech newsletters I am subscribed to. In random order: -* The Imperfectionist -* Register Spill -* Applied Go Weekly Newsletter +* Changelog News * The Valuable Dev * byteSizeGo +* VK Newsletter * Ruby Weekly +* Monospace Mentor * Golang Weekly * The Pragmatic Engineer -* VK Newsletter -* Changelog News * Andreas Brandhorst Newsletter (Sci-Fi author) -* Monospace Mentor +* Register Spill +* Applied Go Weekly Newsletter +* The Imperfectionist ## Magazines I like(d) This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order: -* freeX (not published anymore) -* Linux Magazine * Linux User * LWN (online only) +* freeX (not published anymore) +* Linux Magazine # Formal education diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi index c093386f..512657e3 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi @@ -1,6 +1,6 @@ # f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs -> Published at 2025-04-04T23:21:01+03:00 +> Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025 This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines. @@ -40,6 +40,13 @@ This is the fourth blog post about the f3s series for self-hosting demands in a * ⇢ ⇢ ⇢ FreeBSD host `ubench` benchmark * ⇢ ⇢ ⇢ FreeBSD VM @ Bhyve `ubench` benchmark * ⇢ ⇢ ⇢ Rocky Linux VM @ Bhyve `ubench` benchmark +* ⇢ ⇢ Update: Improving Disk I/O Performance for etcd +* ⇢ ⇢ ⇢ The Problem +* ⇢ ⇢ ⇢ The Solution: Switch to NVMe Emulation +* ⇢ ⇢ ⇢ Step 1: Prepare the Guest OS +* ⇢ ⇢ ⇢ Step 2: Update the Bhyve Configuration +* ⇢ ⇢ ⇢ Benchmark Results +* ⇢ ⇢ ⇢ Important Notes * ⇢ ⇢ Conclusion ## Introduction @@ -498,6 +505,97 @@ Overall, Bhyve has a small overhead, but the CPU performance difference is negli Unfortunately, I wasn't able to find `ubench` in any of the Rocky Linux repositories. So, I skipped this test. +## Update: Improving Disk I/O Performance for etcd + +> Updated: Fri 26 Dec 08:51:23 EET 2025 + +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default `virtio-blk` disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation. + +### The Problem + +The k3s logs showed etcd struggling with disk I/O: + +``` +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"} +{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"} +``` + +A simple sync write benchmark confirmed the issue: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 31.7058 s, 258 kB/s +``` + +### The Solution: Switch to NVMe Emulation + +Bhyve's NVMe emulation provides significantly better I/O performance than `virtio-blk`. + +### Step 1: Prepare the Guest OS + +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation): + +```sh +[root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF +add_drivers+=" nvme nvme_core " +hostonly=no +EOF + +[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf +[root@r0 ~]# dracut -f +[root@r0 ~]# shutdown -h now +``` + +The `hostonly=no` setting ensures the initramfs includes drivers for hardware not currently present. The `use_devicesfile = 0` tells LVM to scan all block devices rather than only those recorded in `/etc/lvm/devices/system.devices` - this is important because the device path changes from `/dev/vda` to `/dev/nvme0n1`. + +### Step 2: Update the Bhyve Configuration + +On the FreeBSD host, update the VM configuration to use NVMe: + +```sh +paul@f0:~ % doas vm stop rocky +paul@f0:~ % doas vm configure rocky +``` + +Change `disk0_type` from `virtio-blk` to `nvme`: + +``` +disk0_type="nvme" +``` + +Then start the VM: + +```sh +paul@f0:~ % doas vm start rocky +``` + +### Benchmark Results + +After switching to NVMe emulation, the sync write performance improved dramatically: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 0.330718 s, 24.8 MB/s +``` + +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s). + +The etcd metrics also showed healthy fsync latencies: + +``` +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408 +``` + +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections. + +### Important Notes + +* Do NOT use `disk0_opts="nocache,direct"` with NVMe emulation - in my testing this actually made performance worse. +* The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot. +* LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path. + ## Conclusion Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD. diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl index b8428906..0cad941c 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi.tpl @@ -1,6 +1,6 @@ # f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs -> Published at 2025-04-04T23:21:01+03:00 +> Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025 This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines. @@ -466,6 +466,97 @@ Overall, Bhyve has a small overhead, but the CPU performance difference is negli Unfortunately, I wasn't able to find `ubench` in any of the Rocky Linux repositories. So, I skipped this test. +## Update: Improving Disk I/O Performance for etcd + +> Updated: Fri 26 Dec 08:51:23 EET 2025 + +After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default `virtio-blk` disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation. + +### The Problem + +The k3s logs showed etcd struggling with disk I/O: + +``` +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"} +{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"} +``` + +A simple sync write benchmark confirmed the issue: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 31.7058 s, 258 kB/s +``` + +### The Solution: Switch to NVMe Emulation + +Bhyve's NVMe emulation provides significantly better I/O performance than `virtio-blk`. + +### Step 1: Prepare the Guest OS + +Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation): + +```sh +[root@r0 ~]# cat > /etc/dracut.conf.d/nvme.conf << EOF +add_drivers+=" nvme nvme_core " +hostonly=no +EOF + +[root@r0 ~]# sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf +[root@r0 ~]# dracut -f +[root@r0 ~]# shutdown -h now +``` + +The `hostonly=no` setting ensures the initramfs includes drivers for hardware not currently present. The `use_devicesfile = 0` tells LVM to scan all block devices rather than only those recorded in `/etc/lvm/devices/system.devices` - this is important because the device path changes from `/dev/vda` to `/dev/nvme0n1`. + +### Step 2: Update the Bhyve Configuration + +On the FreeBSD host, update the VM configuration to use NVMe: + +```sh +paul@f0:~ % doas vm stop rocky +paul@f0:~ % doas vm configure rocky +``` + +Change `disk0_type` from `virtio-blk` to `nvme`: + +``` +disk0_type="nvme" +``` + +Then start the VM: + +```sh +paul@f0:~ % doas vm start rocky +``` + +### Benchmark Results + +After switching to NVMe emulation, the sync write performance improved dramatically: + +```sh +[root@r0 ~]# dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync +8192000 bytes copied, 0.330718 s, 24.8 MB/s +``` + +That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s). + +The etcd metrics also showed healthy fsync latencies: + +``` +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408 +``` + +Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections. + +### Important Notes + +* Do NOT use `disk0_opts="nocache,direct"` with NVMe emulation - in my testing this actually made performance worse. +* The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot. +* LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path. + ## Conclusion Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD. diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index d225500a..89bf6a50 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> - <updated>2025-12-26T01:27:25+02:00</updated> + <updated>2025-12-26T08:51:40+02:00</updated> <title>foo.zone feed</title> <subtitle>To be in the .zone!</subtitle> <link href="gemini://foo.zone/gemfeed/atom.xml" rel="self" /> @@ -9601,7 +9601,7 @@ __ejm\___/________dwb`---`______________________ <title>f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</title> <link href="gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi" /> <id>gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi</id> - <updated>2025-04-04T23:21:01+03:00</updated> + <updated>2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025</updated> <author> <name>Paul Buetow aka snonux</name> <email>paul@dev.buetow.org</email> @@ -9611,7 +9611,7 @@ __ejm\___/________dwb`---`______________________ <div xmlns="http://www.w3.org/1999/xhtml"> <h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-4-rocky-linux-bhyve-vms'>f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</h1><br /> <br /> -<span class='quote'>Published at 2025-04-04T23:21:01+03:00</span><br /> +<span class='quote'>Published at 2025-04-04T23:21:01+03:00, updated Fri 26 Dec 08:51:06 EET 2025</span><br /> <br /> <span>This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.</span><br /> <br /> @@ -9652,6 +9652,13 @@ __ejm\___/________dwb`---`______________________ <li>⇢ ⇢ <a href='#freebsd-host-ubench-benchmark'>FreeBSD host <span class='inlinecode'>ubench</span> benchmark</a></li> <li>⇢ ⇢ <a href='#freebsd-vm--bhyve-ubench-benchmark'>FreeBSD VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</a></li> <li>⇢ ⇢ <a href='#rocky-linux-vm--bhyve-ubench-benchmark'>Rocky Linux VM @ Bhyve <span class='inlinecode'>ubench</span> benchmark</a></li> +<li>⇢ <a href='#update-improving-disk-io-performance-for-etcd'>Update: Improving Disk I/O Performance for etcd</a></li> +<li>⇢ ⇢ <a href='#the-problem'>The Problem</a></li> +<li>⇢ ⇢ <a href='#the-solution-switch-to-nvme-emulation'>The Solution: Switch to NVMe Emulation</a></li> +<li>⇢ ⇢ <a href='#step-1-prepare-the-guest-os'>Step 1: Prepare the Guest OS</a></li> +<li>⇢ ⇢ <a href='#step-2-update-the-bhyve-configuration'>Step 2: Update the Bhyve Configuration</a></li> +<li>⇢ ⇢ <a href='#benchmark-results'>Benchmark Results</a></li> +<li>⇢ ⇢ <a href='#important-notes'>Important Notes</a></li> <li>⇢ <a href='#conclusion'>Conclusion</a></li> </ul><br /> <h2 style='display: inline' id='introduction'>Introduction</h2><br /> @@ -10185,6 +10192,113 @@ Apr <font color="#000000">4</font> <font color="#000000">23</font>:<font color= <br /> <span>Unfortunately, I wasn't able to find <span class='inlinecode'>ubench</span> in any of the Rocky Linux repositories. So, I skipped this test.</span><br /> <br /> +<h2 style='display: inline' id='update-improving-disk-io-performance-for-etcd'>Update: Improving Disk I/O Performance for etcd</h2><br /> +<br /> +<span class='quote'>Updated: Fri 26 Dec 08:51:23 EET 2025</span><br /> +<br /> +<span>After running k3s for some time, I noticed frequent etcd leader elections and "apply request took too long" warnings in the logs. Investigation revealed that etcd's sync writes were extremely slow - around 250 kB/s with the default <span class='inlinecode'>virtio-blk</span> disk emulation. etcd requires fast sync writes (ideally under 10ms fsync latency) for stable operation.</span><br /> +<br /> +<h3 style='display: inline' id='the-problem'>The Problem</h3><br /> +<br /> +<span>The k3s logs showed etcd struggling with disk I/O:</span><br /> +<br /> +<pre> +{"level":"warn","msg":"apply request took too long","took":"4.996516657s","expected-duration":"100ms"} +{"level":"warn","msg":"slow fdatasync","took":"1.328469363s","expected-duration":"1s"} +</pre> +<br /> +<span>A simple sync write benchmark confirmed the issue:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>[root@r0 ~]<i><font color="silver"># dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync</font></i> +<font color="#000000">8192000</font> bytes copied, <font color="#000000">31.7058</font> s, <font color="#000000">258</font> kB/s +</pre> +<br /> +<h3 style='display: inline' id='the-solution-switch-to-nvme-emulation'>The Solution: Switch to NVMe Emulation</h3><br /> +<br /> +<span>Bhyve's NVMe emulation provides significantly better I/O performance than <span class='inlinecode'>virtio-blk</span>.</span><br /> +<br /> +<h3 style='display: inline' id='step-1-prepare-the-guest-os'>Step 1: Prepare the Guest OS</h3><br /> +<br /> +<span>Before changing the disk type, the guest needs NVMe drivers in the initramfs and LVM must be configured to scan all devices (not just those recorded during installation):</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>[root@r0 ~]<i><font color="silver"># cat > /etc/dracut.conf.d/nvme.conf << EOF</font></i> +add_drivers+=<font color="#808080">" nvme nvme_core "</font> +hostonly=no +EOF + +[root@r0 ~]<i><font color="silver"># sed -i 's/# use_devicesfile = 1/use_devicesfile = 0/' /etc/lvm/lvm.conf</font></i> +[root@r0 ~]<i><font color="silver"># dracut -f</font></i> +[root@r0 ~]<i><font color="silver"># shutdown -h now</font></i> +</pre> +<br /> +<span>The <span class='inlinecode'>hostonly=no</span> setting ensures the initramfs includes drivers for hardware not currently present. The <span class='inlinecode'>use_devicesfile = 0</span> tells LVM to scan all block devices rather than only those recorded in <span class='inlinecode'>/etc/lvm/devices/system.devices</span> - this is important because the device path changes from <span class='inlinecode'>/dev/vda</span> to <span class='inlinecode'>/dev/nvme0n1</span>.</span><br /> +<br /> +<h3 style='display: inline' id='step-2-update-the-bhyve-configuration'>Step 2: Update the Bhyve Configuration</h3><br /> +<br /> +<span>On the FreeBSD host, update the VM configuration to use NVMe:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas vm stop rocky +paul@f0:~ % doas vm configure rocky +</pre> +<br /> +<span>Change <span class='inlinecode'>disk0_type</span> from <span class='inlinecode'>virtio-blk</span> to <span class='inlinecode'>nvme</span>:</span><br /> +<br /> +<pre> +disk0_type="nvme" +</pre> +<br /> +<span>Then start the VM:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas vm start rocky +</pre> +<br /> +<h3 style='display: inline' id='benchmark-results'>Benchmark Results</h3><br /> +<br /> +<span>After switching to NVMe emulation, the sync write performance improved dramatically:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>[root@r0 ~]<i><font color="silver"># dd if=/dev/zero of=/tmp/test bs=4k count=2000 oflag=dsync</font></i> +<font color="#000000">8192000</font> bytes copied, <font color="#000000">0.330718</font> s, <font color="#000000">24.8</font> MB/s +</pre> +<br /> +<span>That's approximately **100x faster** than before (24.8 MB/s vs 258 kB/s).</span><br /> +<br /> +<span>The etcd metrics also showed healthy fsync latencies:</span><br /> +<br /> +<pre> +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 347 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 396 +etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408 +</pre> +<br /> +<span>Most fsyncs now complete in under 1ms, and there are no more "slow fdatasync" warnings in the logs. The k3s cluster is now stable without spurious leader elections.</span><br /> +<br /> +<h3 style='display: inline' id='important-notes'>Important Notes</h3><br /> +<br /> +<ul> +<li>Do NOT use <span class='inlinecode'>disk0_opts="nocache,direct"</span> with NVMe emulation - in my testing this actually made performance worse.</li> +<li>The guest OS must have NVMe drivers in the initramfs before switching, otherwise it won't boot.</li> +<li>LVM's devices file feature (enabled by default in RHEL 9 / Rocky Linux 9) must be disabled to allow booting from a different device path.</li> +</ul><br /> <h2 style='display: inline' id='conclusion'>Conclusion</h2><br /> <br /> <span>Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future f3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes, eBPF, systemd) in the Linux world while keeping the steady reliability of FreeBSD.</span><br /> @@ -1,6 +1,6 @@ # Hello! -> This site was generated at 2025-12-26T01:27:25+02:00 by `Gemtexter` +> This site was generated at 2025-12-26T08:51:39+02:00 by `Gemtexter` Welcome to the foo.zone! diff --git a/uptime-stats.gmi b/uptime-stats.gmi index b9efb0e7..d18f38dd 100644 --- a/uptime-stats.gmi +++ b/uptime-stats.gmi @@ -1,6 +1,6 @@ # My machine uptime stats -> This site was last updated at 2025-12-26T01:27:25+02:00 +> This site was last updated at 2025-12-26T08:51:39+02:00 The following stats were collected via `uptimed` on all of my personal computers over many years and the output was generated by `guprecords`, the global uptime records stats analyser of mine. |