diff options
| author | Paul Buetow <paul@buetow.org> | 2025-12-30 10:17:31 +0200 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-12-30 10:17:31 +0200 |
| commit | 4d2e2a870fbf5e3ba60fc88f6c7a394baf8ae6db (patch) | |
| tree | 8766e539fd43348d113fda94f56e15463f29c658 | |
| parent | 105ac52e715728f0fe5b970a0a7a52059f66915e (diff) | |
Update content for html
| -rw-r--r-- | about/resources.html | 202 | ||||
| -rw-r--r-- | gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html | 171 | ||||
| -rw-r--r-- | gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-8b.html | 1132 | ||||
| -rw-r--r-- | gemfeed/atom.xml | 175 | ||||
| -rw-r--r-- | index.html | 2 | ||||
| -rw-r--r-- | tags/style-override.css | 0 | ||||
| -rw-r--r-- | test-tags.html.tmp | 38 | ||||
| -rw-r--r-- | test-template-tags.html.tmp | 38 | ||||
| -rw-r--r-- | uptime-stats.html | 90 |
9 files changed, 1635 insertions, 213 deletions
diff --git a/about/resources.html b/about/resources.html index 417c5142..ec883081 100644 --- a/about/resources.html +++ b/about/resources.html @@ -50,67 +50,67 @@ <span>In random order:</span><br /> <br /> <ul> -<li>Funktionale Programmierung; Peter Pepper; Springer</li> -<li>Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press</li> -<li>The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional</li> -<li>Raku Fundamentals; Moritz Lenz; Apress</li> -<li>Raku Recipes; J.J. Merelo; Apress</li> -<li>Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner</li> -<li>Polished Ruby Programming; Jeremy Evans; Packt Publishing</li> -<li>Modern Perl; Chromatic ; Onyx Neon Press</li> -<li>Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers </li> -<li>Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson</li> -<li>DNS and BIND; Cricket Liu; O'Reilly</li> -<li>100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications</li> -<li>Effective awk programming; Arnold Robbins; O'Reilly</li> -<li>The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress</li> <li>Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly</li> -<li>Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf</li> -<li>Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications</li> -<li>The Docker Book; James Turnbull; Kindle</li> -<li>Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly</li> <li>The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible</li> -<li>Data Science at the Command Line; Jeroen Janssens; O'Reilly</li> -<li>Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook</li> +<li>Effective awk programming; Arnold Robbins; O'Reilly</li> +<li>Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly</li> +<li>Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers </li> +<li>Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt </li> +<li>Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf</li> +<li>Leanring eBPF; Liz Rice; O'Reilly</li> +<li>97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly</li> <li>Effective Java; Joshua Bloch; Addison-Wesley Professional</li> -<li>Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly</li> +<li>The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress</li> +<li>Raku Fundamentals; Moritz Lenz; Apress</li> +<li>C++ Programming Language; Bjarne Stroustrup;</li> +<li>The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional</li> +<li>100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications</li> +<li>Java ist auch eine Insel; Christian Ullenboom; </li> +<li>Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers</li> <li>Ultimate Go Notebook; Bill Kennedy</li> -<li>Perl New Features; Joshua McAdams, brian d foy; Perl School</li> -<li>Higher Order Perl; Mark Dominus; Morgan Kaufmann</li> -<li>Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook</li> -<li>Terraform Cookbook; Mikael Krief; Packt Publishing</li> -<li>Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly</li> -<li>Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly</li> -<li>The Pragmatic Programmer; David Thomas; Addison-Wesley</li> <li>Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press</li> -<li>Developing Games in Java; David Brackeen and others...; New Riders</li> +<li>Data Science at the Command Line; Jeroen Janssens; O'Reilly</li> +<li>Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson</li> <li>DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible</li> +<li>Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook</li> +<li>Developing Games in Java; David Brackeen and others...; New Riders</li> +<li>Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly</li> +<li>Raku Recipes; J.J. Merelo; Apress</li> <li>Site Reliability Engineering; How Google runs production systems; O'Reilly</li> -<li>97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly</li> +<li>The Docker Book; James Turnbull; Kindle</li> +<li>Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook</li> +<li>Polished Ruby Programming; Jeremy Evans; Packt Publishing</li> +<li>Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications</li> <li>Pro Puppet; James Turnbull, Jeffrey McCune; Apress</li> -<li>Java ist auch eine Insel; Christian Ullenboom; </li> +<li>Systemprogrammierung in Go; Frank Müller; dpunkt</li> +<li>Terraform Cookbook; Mikael Krief; Packt Publishing</li> +<li>Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner</li> +<li>Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly</li> +<li>Higher Order Perl; Mark Dominus; Morgan Kaufmann</li> +<li>Perl New Features; Joshua McAdams, brian d foy; Perl School</li> +<li>Funktionale Programmierung; Peter Pepper; Springer</li> +<li>Modern Perl; Chromatic ; Onyx Neon Press</li> <li>Concurrency in Go; Katherine Cox-Buday; O'Reilly</li> -<li>Leanring eBPF; Liz Rice; O'Reilly</li> -<li>Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers</li> <li>The Kubernetes Book; Nigel Poulton; Unabridged Audiobook</li> -<li>C++ Programming Language; Bjarne Stroustrup;</li> -<li>The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton</li> -<li>Systemprogrammierung in Go; Frank Müller; dpunkt</li> -<li>Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt </li> +<li>DNS and BIND; Cricket Liu; O'Reilly</li> <li>21st Century C: C Tips from the New School; Ben Klemens; O'Reilly</li> +<li>Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly</li> +<li>The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton</li> +<li>The Pragmatic Programmer; David Thomas; Addison-Wesley</li> +<li>Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press</li> </ul><br /> <h2 style='display: inline' id='technical-references'>Technical references</h2><br /> <br /> <span>I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order:</span><br /> <br /> <ul> +<li>Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly</li> <li>Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley</li> -<li>Go: Design Patterns for Real-World Projects; Mat Ryer; Packt</li> +<li>Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly</li> <li>The Linux Programming Interface; Michael Kerrisk; No Starch Press </li> -<li>Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly</li> <li>Relayd and Httpd Mastery; Michael W Lucas</li> <li>BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley</li> -<li>Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly</li> +<li>Go: Design Patterns for Real-World Projects; Mat Ryer; Packt</li> <li>Implementing Service Level Objectives; Alex Hidalgo; O'Reilly</li> </ul><br /> <h2 style='display: inline' id='self-development-and-soft-skills-books'>Self-development and soft-skills books</h2><br /> @@ -118,44 +118,44 @@ <span>In random order:</span><br /> <br /> <ul> -<li>The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook </li> -<li>Psycho-Cybernetics; Maxwell Maltz; Perigee Books</li> <li>Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook</li> -<li>The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook</li> -<li>Eat That Frog; Brian Tracy</li> -<li>The Bullet Journal Method; Ryder Carroll; Fourth Estate</li> -<li>Slow Productivity; Cal Newport; Penguin Random House</li> +<li>Soft Skills; John Sommez; Manning Publications</li> +<li>Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press</li> +<li>Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing</li> <li>Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook</li> -<li>The Good Enough Job; Simone Stolzoff; Ebury Edge</li> -<li>Eat That Frog!; Brian Tracy; Hodder Paperbacks</li> -<li>The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd</li> <li>97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook</li> -<li>The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME)</li> +<li>The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select</li> +<li>101 Essays that change the way you think; Brianna Wiest; Audiobook</li> +<li>Psycho-Cybernetics; Maxwell Maltz; Perigee Books</li> <li>The Power of Now; Eckhard Tolle; Yellow Kite</li> -<li>Soft Skills; John Sommez; Manning Publications</li> -<li>Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne</li> +<li>Influence without Authority; A. Cohen, D. Bradford; Wiley</li> +<li>Ultralearning; Anna Laurent; Self-published via Amazon</li> <li>Solve for Happy; Mo Gawdat (RE-READ 1ST TIME)</li> -<li>The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select</li> -<li>The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook</li> -<li>Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press</li> +<li>The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books</li> <li>Atomic Habits; James Clear; Random House Business</li> -<li>Never Split the Difference; Chris Voss, Tahl Raz; Random House Business</li> -<li>Getting Things Done; David Allen</li> <li>So Good They Can't Ignore You; Cal Newport; Business Plus</li> -<li>Ultralearning; Anna Laurent; Self-published via Amazon</li> -<li>The Joy of Missing Out; Christina Crook; New Society Publishers</li> -<li>Digital Minimalism; Cal Newport; Portofolio Penguin</li> -<li>Ultralearning; Scott Young; Thorsons</li> -<li>Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion</li> -<li>101 Essays that change the way you think; Brianna Wiest; Audiobook</li> -<li>Influence without Authority; A. Cohen, D. Bradford; Wiley</li> <li>Stop starting, start finishing; Arne Roock; Lean-Kanban University </li> +<li>Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne</li> +<li>The Joy of Missing Out; Christina Crook; New Society Publishers</li> <li>The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK</li> +<li>Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion</li> +<li>The Good Enough Job; Simone Stolzoff; Ebury Edge</li> +<li>Ultralearning; Scott Young; Thorsons</li> +<li>Digital Minimalism; Cal Newport; Portofolio Penguin</li> +<li>The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd</li> +<li>The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook </li> <li>Deep Work; Cal Newport; Piatkus</li> -<li>The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books</li> -<li>Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing</li> -<li>Meditation for Mortals, Oliver Burkeman, Audiobook</li> +<li>Eat That Frog; Brian Tracy</li> +<li>The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME)</li> <li>Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly</li> +<li>Eat That Frog!; Brian Tracy; Hodder Paperbacks</li> +<li>Getting Things Done; David Allen</li> +<li>The Bullet Journal Method; Ryder Carroll; Fourth Estate</li> +<li>The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook</li> +<li>Never Split the Difference; Chris Voss, Tahl Raz; Random House Business</li> +<li>The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook</li> +<li>Meditation for Mortals, Oliver Burkeman, Audiobook</li> +<li>Slow Productivity; Cal Newport; Penguin Random House</li> </ul><br /> <a class='textlink' href='../notes/index.html'>Here are notes of mine for some of the books</a><br /> <br /> @@ -165,30 +165,30 @@ <br /> <ul> <li>Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online</li> -<li>MySQL Deep Dive Workshop; 2-day on-site training</li> -<li>AWS Immersion Day; Amazon; 1-day interactive online training </li> +<li>F5 Loadbalancers Training; 2-day on-site training; F5, Inc. </li> <li>The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online</li> -<li>Protocol buffers; O'Reilly Online</li> -<li>The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online</li> -<li>Scripting Vim; Damian Conway; O'Reilly Online</li> -<li>Developing IaC with Terraform (with Live Lessons); O'Reilly Online</li> -<li>Structure and Interpretation of Computer Programs; Harold Abelson and more...; </li> -<li>Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need)</li> <li>Ultimate Go Programming; Bill Kennedy; O'Reilly Online</li> -<li>Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training</li> +<li>Structure and Interpretation of Computer Programs; Harold Abelson and more...; </li> <li>Apache Tomcat Best Practises; 3-day on-site training</li> -<li>F5 Loadbalancers Training; 2-day on-site training; F5, Inc. </li> <li>Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon</li> +<li>The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online</li> +<li>Protocol buffers; O'Reilly Online</li> +<li>Scripting Vim; Damian Conway; O'Reilly Online</li> +<li>MySQL Deep Dive Workshop; 2-day on-site training</li> +<li>Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training</li> +<li>Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need)</li> <li>Functional programming lecture; Remote University of Hagen</li> +<li>AWS Immersion Day; Amazon; 1-day interactive online training </li> +<li>Developing IaC with Terraform (with Live Lessons); O'Reilly Online</li> </ul><br /> <h2 style='display: inline' id='technical-guides'>Technical guides</h2><br /> <br /> <span>These are not whole books, but guides (smaller or larger) which I found very useful. in random order:</span><br /> <br /> <ul> +<li>Advanced Bash-Scripting Guide </li> <li>Raku Guide at https://raku.guide </li> <li>How CPUs work at https://cpu.land</li> -<li>Advanced Bash-Scripting Guide </li> </ul><br /> <h2 style='display: inline' id='podcasts'>Podcasts</h2><br /> <br /> @@ -197,32 +197,32 @@ <span>In random order:</span><br /> <br /> <ul> -<li>Wednesday Wisdom</li> -<li>Modern Mentor</li> -<li>The Pragmatic Engineer Podcast</li> -<li>The Changelog Podcast(s)</li> -<li>BSD Now [BSD]</li> -<li>Fork Around And Find Out</li> -<li>Dev Interrupted</li> <li>Cup o' Go [Golang]</li> -<li>Fallthrough [Golang]</li> +<li>Dev Interrupted</li> +<li>Fork Around And Find Out</li> +<li>Maintainable</li> +<li>BSD Now [BSD]</li> +<li>The Changelog Podcast(s)</li> <li>Hidden Brain</li> +<li>The Pragmatic Engineer Podcast</li> +<li>Fallthrough [Golang]</li> <li>The ProdCast (Google SRE Podcast)</li> -<li>Pratical AI</li> -<li>Backend Banter</li> <li>Deep Questions with Cal Newport</li> -<li>Maintainable</li> +<li>Backend Banter</li> +<li>Wednesday Wisdom</li> +<li>Pratical AI</li> +<li>Modern Mentor</li> </ul><br /> <h3 style='display: inline' id='podcasts-i-liked'>Podcasts I liked</h3><br /> <br /> <span>I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests.</span><br /> <br /> <ul> -<li>FLOSS weekly</li> -<li>Go Time (predecessor of fallthrough)</li> +<li>Ship It (predecessor of Fork Around And Find Out)</li> <li>Modern Mentor</li> <li>CRE: Chaosradio Express [german]</li> -<li>Ship It (predecessor of Fork Around And Find Out)</li> +<li>Go Time (predecessor of fallthrough)</li> +<li>FLOSS weekly</li> <li>Java Pub House</li> </ul><br /> <h2 style='display: inline' id='newsletters-i-like'>Newsletters I like</h2><br /> @@ -230,26 +230,26 @@ <span>This is a mix of tech and non-tech newsletters I am subscribed to. In random order:</span><br /> <br /> <ul> -<li>VK Newsletter</li> -<li>Register Spill</li> -<li>Monospace Mentor</li> <li>Andreas Brandhorst Newsletter (Sci-Fi author)</li> <li>Applied Go Weekly Newsletter</li> -<li>The Imperfectionist</li> -<li>Changelog News</li> -<li>The Valuable Dev</li> +<li>Ruby Weekly</li> +<li>The Pragmatic Engineer</li> +<li>Register Spill</li> +<li>VK Newsletter</li> <li>byteSizeGo</li> <li>Golang Weekly</li> -<li>The Pragmatic Engineer</li> -<li>Ruby Weekly</li> +<li>Monospace Mentor</li> +<li>Changelog News</li> +<li>The Imperfectionist</li> +<li>The Valuable Dev</li> </ul><br /> <h2 style='display: inline' id='magazines-i-liked'>Magazines I like(d)</h2><br /> <br /> <span>This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order:</span><br /> <br /> <ul> -<li>Linux Magazine</li> <li>LWN (online only)</li> +<li>Linux Magazine</li> <li>Linux User</li> <li>freeX (not published anymore)</li> </ul><br /> diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html index 25d807c2..b4ae12dd 100644 --- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html +++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html @@ -13,7 +13,7 @@ </p> <h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-7-k3s-and-first-pod-deployments'>f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</h1><br /> <br /> -<span class='quote'>Published at 2025-10-02T11:27:19+03:00</span><br /> +<span class='quote'>Published at 2025-10-02T11:27:19+03:00, last updated Tue 30 Dec 10:11:58 EET 2025</span><br /> <br /> <span>This is the seventh blog post about the f3s series for my self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.</span><br /> <br /> @@ -43,6 +43,8 @@ <li>⇢ ⇢ <a href='#scaling-traefik-for-faster-failover'>Scaling Traefik for faster failover</a></li> <li>⇢ <a href='#make-it-accessible-from-the-public-internet'>Make it accessible from the public internet</a></li> <li>⇢ ⇢ <a href='#openbsd-relayd-configuration'>OpenBSD relayd configuration</a></li> +<li>⇢ ⇢ <a href='#automatic-failover-when-f3s-cluster-is-down'>Automatic failover when f3s cluster is down</a></li> +<li>⇢ ⇢ <a href='#openbsd-httpd-fallback-configuration'>OpenBSD httpd fallback configuration</a></li> <li>⇢ <a href='#deploying-the-private-docker-image-registry'>Deploying the private Docker image registry</a></li> <li>⇢ ⇢ <a href='#prepare-the-nfs-backed-storage'>Prepare the NFS-backed storage</a></li> <li>⇢ ⇢ <a href='#install-or-upgrade-the-chart'>Install (or upgrade) the chart</a></li> @@ -672,10 +674,11 @@ table <f3s> { } </pre> <br /> -<span>Inside the <span class='inlinecode'>http protocol "https"</span> block each public hostname gets its Let's Encrypt certificate and is matched to that backend table. Besides the primary trio, every service-specific hostname (<span class='inlinecode'>anki</span>, <span class='inlinecode'>bag</span>, <span class='inlinecode'>flux</span>, <span class='inlinecode'>audiobookshelf</span>, <span class='inlinecode'>gpodder</span>, <span class='inlinecode'>radicale</span>, <span class='inlinecode'>vault</span>, <span class='inlinecode'>syncthing</span>, <span class='inlinecode'>uprecords</span>) and their <span class='inlinecode'>www</span> / <span class='inlinecode'>standby</span> aliases reuse the same pool so new apps can go live just by publishing an ingress rule, whereas they will all map to a service running in k3s:</span><br /> +<span>Inside the <span class='inlinecode'>http protocol "https"</span> block each public hostname gets its Let's Encrypt certificate. The protocol configures TLS keypairs for all f3s services and other public endpoints. For f3s hosts specifically, there are no explicit <span class='inlinecode'>forward to</span> rules in the protocol—they use the relay-level failover mechanism described later. Non-f3s hosts get explicit localhost routing to prevent them from trying the f3s backends:</span><br /> <br /> <pre> http protocol "https" { + # TLS certificates for all f3s services tls keypair f3s.foo.zone tls keypair www.f3s.foo.zone tls keypair standby.f3s.foo.zone @@ -707,36 +710,15 @@ http protocol "https" { tls keypair www.uprecords.f3s.foo.zone tls keypair standby.uprecords.f3s.foo.zone - match request quick header "Host" value "f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "anki.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.anki.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.anki.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "bag.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.bag.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.bag.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "flux.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.flux.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.flux.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "audiobookshelf.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.audiobookshelf.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.audiobookshelf.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "gpodder.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.gpodder.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.gpodder.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "radicale.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.radicale.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.radicale.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "vault.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.vault.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.vault.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "syncthing.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.syncthing.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.syncthing.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "uprecords.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.uprecords.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.uprecords.f3s.foo.zone" forward to <f3s> + # Explicitly route non-f3s hosts to localhost + match request header "Host" value "foo.zone" forward to <localhost> + match request header "Host" value "www.foo.zone" forward to <localhost> + match request header "Host" value "dtail.dev" forward to <localhost> + # ... other non-f3s hosts ... + + # NOTE: f3s hosts have NO match rules here! + # They use relay-level failover (f3s -> localhost backup) + # See the relay configuration below for automatic failover details } </pre> <br /> @@ -746,18 +728,143 @@ http protocol "https" { relay "https4" { listen on 46.23.94.99 port 443 tls protocol "https" + # Primary: f3s cluster (with health checks) - Falls back to localhost when all hosts down forward to <f3s> port 80 check tcp + forward to <localhost> port 8080 } relay "https6" { listen on 2a03:6000:6f67:624::99 port 443 tls protocol "https" + # Primary: f3s cluster (with health checks) - Falls back to localhost when all hosts down forward to <f3s> port 80 check tcp + forward to <localhost> port 8080 } </pre> <br /> <span>In practice, that means relayd terminates TLS with the correct certificate, keeps the three WireGuard-connected backends in rotation, and ships each request to whichever bhyve VM answers first.</span><br /> <br /> +<h3 style='display: inline' id='automatic-failover-when-f3s-cluster-is-down'>Automatic failover when f3s cluster is down</h3><br /> +<br /> +<span class='quote'>Update: This section was added at Tue 30 Dec 10:11:44 EET 2025</span><br /> +<br /> +<span>One important aspect of this setup is graceful degradation: when all three f3s nodes are unreachable (e.g., during maintenance or a power outage in my LAN), users should see a friendly status page instead of an error message.</span><br /> +<br /> +<span>OpenBSD's relayd supports automatic failover through its health check mechanism. According to the relayd.conf manual:</span><br /> +<br /> +<span class='quote'>This directive can be specified multiple times - subsequent entries will be used as the backup table if all hosts in the previous table are down.</span><br /> +<br /> +<span>The key is the order of <span class='inlinecode'>forward to</span> statements in the relay configuration. By placing the f3s table first with <span class='inlinecode'>check tcp</span> health checks, followed by localhost as a backup, relayd automatically routes traffic based on backend availability:</span><br /> +<br /> +<span>When f3s cluster is UP:</span><br /> +<br /> +<ul> +<li>Health checks on port 80 succeed for f3s nodes</li> +<li>All f3s traffic routes to the Kubernetes cluster</li> +<li>Localhost backup remains idle</li> +</ul><br /> +<span>When f3s cluster is DOWN:</span><br /> +<br /> +<ul> +<li>All health checks fail (nodes unreachable)</li> +<li>The <span class='inlinecode'><f3s></span> table becomes unavailable</li> +<li>Traffic automatically falls back to <span class='inlinecode'><localhost></span> on port 8080</li> +<li>OpenBSD's httpd serves a static fallback page</li> +</ul><br /> +<pre> +# NEW configuration - supports automatic failover +http protocol "https" { + # Explicitly route non-f3s hosts to localhost + match request header "Host" value "foo.zone" forward to <localhost> + match request header "Host" value "dtail.dev" forward to <localhost> + # ... other non-f3s hosts ... + + # f3s hosts have NO protocol rules - they use relay-level failover + # (no match rules for f3s.foo.zone, anki.f3s.foo.zone, etc.) +} + +relay "https4" { + # f3s FIRST (with health checks), localhost as BACKUP + forward to <f3s> port 80 check tcp + forward to <localhost> port 8080 +} +</pre> +<br /> +<span>This way, f3s traffic uses the relay's default behavior: try the first table, fall back to the second when health checks fail.</span><br /> +<br /> +<h3 style='display: inline' id='openbsd-httpd-fallback-configuration'>OpenBSD httpd fallback configuration</h3><br /> +<br /> +<span>The localhost httpd service on port 8080 serves the fallback content from <span class='inlinecode'>/var/www/htdocs/f3s_fallback/</span>. This directory contains a simple HTML page explaining the situation:</span><br /> +<br /> +<pre> +# OpenBSD httpd.conf +# Fallback for f3s hosts +server "f3s.foo.zone" { + listen on * port 8080 + log style forwarded + location * { + root "/htdocs/f3s_fallback" + directory auto index + } +} + +server "anki.f3s.foo.zone" { + listen on * port 8080 + log style forwarded + location * { + root "/htdocs/f3s_fallback" + directory auto index + } +} + +# ... similar blocks for all f3s hostnames ... +</pre> +<br /> +<span>The fallback page itself is straightforward:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><b><u><font color="#000000"><!DOCTYPE</font></u></b> <b><font color="#000000">html</font></b><b><u><font color="#000000">></font></u></b> +<b><u><font color="#000000"><html></font></u></b> +<b><u><font color="#000000"><head></font></u></b> + <b><u><font color="#000000"><title></font></u></b>Server turned off<b><u><font color="#000000"></title></font></u></b> + <b><u><font color="#000000"><style></font></u></b> + body { + font-family: <font color="#808080">sans-serif</font>; + text-align: <font color="#808080">center</font>; + padding-top: <font color="#808080">50px</font>; + } + .container { + max-width: <font color="#808080">600px</font>; + margin: <font color="#808080">0</font> <font color="#808080">auto</font>; + } + <b><u><font color="#000000"></style></font></u></b> +<b><u><font color="#000000"></head></font></u></b> +<b><u><font color="#000000"><body></font></u></b> + <b><u><font color="#000000"><div</font></u></b> <b><font color="#000000">class</font></b>=<font color="#808080">"container"</font><b><u><font color="#000000">></font></u></b> + <b><u><font color="#000000"><h1></font></u></b>Server turned off<b><u><font color="#000000"></h1></font></u></b> + <b><u><font color="#000000"><p></font></u></b>The servers are all currently turned off.<b><u><font color="#000000"></p></font></u></b> + <b><u><font color="#000000"><p></font></u></b>Please try again later.<b><u><font color="#000000"></p></font></u></b> + <b><u><font color="#000000"><p></font></u></b>Or email <b><u><font color="#000000"><a</font></u></b> <b><font color="#000000">href</font></b>=<font color="#808080">"mailto:paul@nospam.buetow.org"</font><b><u><font color="#000000">></font></u></b>paul@nospam.buetow.org<b><u><font color="#000000"></a></font></u></b> + - so I can turn them back on for you!<b><u><font color="#000000"></p></font></u></b> + <b><u><font color="#000000"></div></font></u></b> +<b><u><font color="#000000"></body></font></u></b> +<b><u><font color="#000000"></html></font></u></b> +</pre> +<br /> +<span>This approach provides several benefits:</span><br /> +<br /> +<ul> +<li>Automatic detection: Health checks run continuously; no manual intervention needed</li> +<li>Instant fallback: When all f3s nodes go down, the next request automatically routes to localhost</li> +<li>Transparent recovery: When f3s comes back online, health checks pass and traffic resumes automatically</li> +<li>User experience: Visitors see a helpful message instead of connection errors</li> +<li>No DNS changes: The same hostnames work whether f3s is up or down</li> +</ul><br /> +<span>This fallback mechanism has proven invaluable during maintenance windows and unexpected outages, ensuring that users always get a response even when the home lab is offline.</span><br /> +<br /> <h2 style='display: inline' id='deploying-the-private-docker-image-registry'>Deploying the private Docker image registry</h2><br /> <br /> <span>As not all Docker images I want to deploy are available on public Docker registries and as I also build some of them by myself, there is the need of a private registry. </span><br /> diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-8b.html b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-8b.html new file mode 100644 index 00000000..d598c631 --- /dev/null +++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-8b.html @@ -0,0 +1,1132 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<title>f3s: Kubernetes with FreeBSD - Part 9: Enabling etcd Metrics</title> +<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> +<link rel="stylesheet" href="../style.css" /> +<link rel="stylesheet" href="style-override.css" /> +</head> +<body> +<p class="header"> +<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-8b.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-8b.gmi">Gemini</a> +</p> +<h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-9-enabling-etcd-metrics'>f3s: Kubernetes with FreeBSD - Part 9: Enabling etcd Metrics</h1><br /> +<br /> +<h2 style='display: inline' id='introduction'>Introduction</h2><br /> +<br /> +<span>This post covers enabling etcd metrics monitoring for the k3s cluster. The etcd dashboard in Grafana initially showed no data because k3s uses an embedded etcd that doesn't expose metrics by default.</span><br /> +<br /> +<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>Part 8: Observability</a><br /> +<br /> +<h2 style='display: inline' id='enabling-etcd-metrics-in-k3s'>Enabling etcd metrics in k3s</h2><br /> +<br /> +<span>On each control-plane node (r0, r1, r2), create /etc/rancher/k3s/config.yaml:</span><br /> +<br /> +<pre> +etcd-expose-metrics: true +</pre> +<br /> +<span>Then restart k3s on each node:</span><br /> +<br /> +<pre> +systemctl restart k3s +</pre> +<br /> +<span>After restarting, etcd metrics are available on port 2381:</span><br /> +<br /> +<pre> +curl http://127.0.0.1:2381/metrics | grep etcd +</pre> +<br /> +<h2 style='display: inline' id='configuring-prometheus-to-scrape-etcd'>Configuring Prometheus to scrape etcd</h2><br /> +<br /> +<span>In persistence-values.yaml, enable kubeEtcd with the node IP addresses:</span><br /> +<br /> +<pre> +kubeEtcd: + enabled: true + endpoints: + - 192.168.1.120 + - 192.168.1.121 + - 192.168.1.122 + service: + enabled: true + port: 2381 + targetPort: 2381 +</pre> +<br /> +<span>Apply the changes:</span><br /> +<br /> +<pre> +just upgrade +</pre> +<br /> +<h2 style='display: inline' id='verifying-etcd-metrics'>Verifying etcd metrics</h2><br /> +<br /> +<span>After the changes, all etcd targets are being scraped:</span><br /> +<br /> +<pre> +kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \ + -c prometheus -- wget -qO- 'http://localhost:9090/api/v1/query?query=etcd_server_has_leader' | \ + jq -r '.data.result[] | "\(.metric.instance): \(.value[1])"' +</pre> +<br /> +<span>Output:</span><br /> +<br /> +<pre> +192.168.1.120:2381: 1 +192.168.1.121:2381: 1 +192.168.1.122:2381: 1 +</pre> +<br /> +<span>The etcd dashboard in Grafana now displays metrics including Raft proposals, leader elections, and peer round trip times.</span><br /> +<br /> +<h2 style='display: inline' id='complete-persistence-valuesyaml'>Complete persistence-values.yaml</h2><br /> +<br /> +<span>The complete updated persistence-values.yaml:</span><br /> +<br /> +<pre> +kubeEtcd: + enabled: true + endpoints: + - 192.168.1.120 + - 192.168.1.121 + - 192.168.1.122 + service: + enabled: true + port: 2381 + targetPort: 2381 + +prometheus: + prometheusSpec: + additionalScrapeConfigsSecret: + enabled: true + name: additional-scrape-configs + key: additional-scrape-configs.yaml + storageSpec: + volumeClaimTemplate: + spec: + storageClassName: "" + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 10Gi + selector: + matchLabels: + type: local + app: prometheus + +grafana: + persistence: + enabled: true + type: pvc + existingClaim: "grafana-data-pvc" + + initChownData: + enabled: false + + podSecurityContext: + fsGroup: 911 + runAsUser: 911 + runAsGroup: 911 +</pre> +<br /> +<h2 style='display: inline' id='zfs-monitoring-for-freebsd-servers'>ZFS Monitoring for FreeBSD Servers</h2><br /> +<br /> +<span>The FreeBSD servers (f0, f1, f2) that provide NFS storage to the k3s cluster have ZFS filesystems. Monitoring ZFS performance is crucial for understanding storage performance and cache efficiency.</span><br /> +<br /> +<h3 style='display: inline' id='node-exporter-zfs-collector'>Node Exporter ZFS Collector</h3><br /> +<br /> +<span>The node_exporter running on each FreeBSD server (v1.9.1) includes a built-in ZFS collector that exposes metrics via sysctls. The ZFS collector is enabled by default and provides:</span><br /> +<br /> +<ul> +<li>ARC (Adaptive Replacement Cache) statistics</li> +<li>Cache hit/miss rates</li> +<li>Memory usage and allocation</li> +<li>MRU/MFU cache breakdown</li> +<li>Data vs metadata distribution</li> +</ul><br /> +<h3 style='display: inline' id='verifying-zfs-metrics'>Verifying ZFS Metrics</h3><br /> +<br /> +<span>On any FreeBSD server, check that ZFS metrics are being exposed:</span><br /> +<br /> +<pre> +paul@f0:~ % curl -s http://localhost:9100/metrics | grep node_zfs_arcstats | wc -l + 69 +</pre> +<br /> +<span>The metrics are automatically scraped by Prometheus through the existing static configuration in additional-scrape-configs.yaml which targets all FreeBSD servers on port 9100 with the os: freebsd label.</span><br /> +<br /> +<h3 style='display: inline' id='zfs-recording-rules'>ZFS Recording Rules</h3><br /> +<br /> +<span>Created recording rules for easier dashboard consumption in zfs-recording-rules.yaml:</span><br /> +<br /> +<pre> +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: freebsd-zfs-rules + namespace: monitoring + labels: + release: prometheus +spec: + groups: + - name: freebsd-zfs-arc + interval: 30s + rules: + - record: node_zfs_arc_hit_rate_percent + expr: | + 100 * ( + rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) / + (rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) + + rate(node_zfs_arcstats_misses_total{os="freebsd"}[5m])) + ) + labels: + os: freebsd + - record: node_zfs_arc_memory_usage_percent + expr: | + 100 * ( + node_zfs_arcstats_size_bytes{os="freebsd"} / + node_zfs_arcstats_c_max_bytes{os="freebsd"} + ) + labels: + os: freebsd + # Additional rules for metadata %, target %, MRU/MFU %, etc. +</pre> +<br /> +<span>These recording rules calculate:</span><br /> +<br /> +<ul> +<li>ARC hit rate percentage</li> +<li>ARC memory usage percentage (current vs maximum)</li> +<li>ARC target percentage (target vs maximum)</li> +<li>Metadata vs data percentages</li> +<li>MRU vs MFU cache percentages</li> +<li>Demand data and metadata hit rates</li> +</ul><br /> +<h3 style='display: inline' id='grafana-dashboards'>Grafana Dashboards</h3><br /> +<br /> +<span>Created two comprehensive ZFS monitoring dashboards (zfs-dashboards.yaml):</span><br /> +<br /> +<span>**Dashboard 1: FreeBSD ZFS (per-host detailed view)**</span><br /> +<br /> +<span>Includes variables to select:</span><br /> +<ul> +<li>FreeBSD server (f0, f1, or f2)</li> +<li>ZFS pool (zdata, zroot, or all)</li> +</ul><br /> +<span>**Pool Overview Row:**</span><br /> +<ul> +<li>Pool Capacity gauge (with thresholds: green <70%, yellow <85%, red >85%)</li> +<li>Pool Health status (ONLINE/DEGRADED/FAULTED with color coding)</li> +<li>Total Pool Size stat</li> +<li>Free Space stat</li> +<li>Pool Space Usage Over Time (stacked: used + free)</li> +<li>Pool Capacity Trend time series</li> +</ul><br /> +<span>**Dataset Statistics Row:**</span><br /> +<ul> +<li>Table showing all datasets with columns: Pool, Dataset, Used, Available, Referenced</li> +<li>Automatically filters by selected pool</li> +</ul><br /> +<span>**ARC Cache Statistics Row:**</span><br /> +<ul> +<li>ARC Hit Rate gauge (red <70%, yellow <90%, green >=90%)</li> +<li>ARC Size time series (current, target, max)</li> +<li>ARC Memory Usage percentage gauge</li> +<li>ARC Hits vs Misses rate</li> +<li>ARC Data vs Metadata stacked time series</li> +</ul><br /> +<span>**Dashboard 2: FreeBSD ZFS Summary (cluster-wide overview)**</span><br /> +<br /> +<span>**Cluster-Wide Pool Statistics Row:**</span><br /> +<ul> +<li>Total Storage Capacity across all servers</li> +<li>Total Used space</li> +<li>Total Free space</li> +<li>Average Pool Capacity gauge</li> +<li>Pool Health Status (worst case across cluster)</li> +<li>Total Pool Space Usage Over Time</li> +<li>Per-Pool Capacity time series (all pools on all hosts)</li> +</ul><br /> +<span>**Per-Host Pool Breakdown Row:**</span><br /> +<ul> +<li>Bar gauge showing capacity by host and pool</li> +<li>Table with all pools: Host, Pool, Size, Used, Free, Capacity %, Health</li> +</ul><br /> +<span>**Cluster-Wide ARC Statistics Row:**</span><br /> +<ul> +<li>Average ARC Hit Rate gauge across all hosts</li> +<li>ARC Hit Rate by Host time series</li> +<li>Total ARC Size Across Cluster</li> +<li>Total ARC Hits vs Misses (cluster-wide sum)</li> +<li>ARC Size by Host</li> +</ul><br /> +<span>**Dashboard Visualization:**</span><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-8b/grafana-zfs-dashboard.png'><img alt='ZFS monitoring dashboard in Grafana showing pool statistics and ARC cache metrics' title='ZFS monitoring dashboard in Grafana showing pool statistics and ARC cache metrics' src='./f3s-kubernetes-with-freebsd-part-8b/grafana-zfs-dashboard.png' /></a><br /> +<br /> +<h3 style='display: inline' id='deployment'>Deployment</h3><br /> +<br /> +<span>Applied the resources to the cluster:</span><br /> +<br /> +<pre> +cd /home/paul/git/conf/f3s/prometheus +kubectl apply -f zfs-recording-rules.yaml +kubectl apply -f zfs-dashboards.yaml +</pre> +<br /> +<span>Updated Justfile to include ZFS recording rules in install and upgrade targets:</span><br /> +<br /> +<pre> +install: + kubectl apply -f persistent-volumes.yaml + kubectl create secret generic additional-scrape-configs --from-file=additional-scrape-configs.yaml -n monitoring --dry-run=client -o yaml | kubectl apply -f - + helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring -f persistence-values.yaml + kubectl apply -f freebsd-recording-rules.yaml + kubectl apply -f openbsd-recording-rules.yaml + kubectl apply -f zfs-recording-rules.yaml + just -f grafana-ingress/Justfile install +</pre> +<br /> +<h3 style='display: inline' id='verifying-zfs-metrics-in-prometheus'>Verifying ZFS Metrics in Prometheus</h3><br /> +<br /> +<span>Check that ZFS metrics are being collected:</span><br /> +<br /> +<pre> +kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \ + wget -qO- 'http://localhost:9090/api/v1/query?query=node_zfs_arcstats_size_bytes' +</pre> +<br /> +<span>Check recording rules are calculating correctly:</span><br /> +<br /> +<pre> +kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c prometheus -- \ + wget -qO- 'http://localhost:9090/api/v1/query?query=node_zfs_arc_memory_usage_percent' +</pre> +<br /> +<span>Example output shows memory usage percentage for each FreeBSD server:</span><br /> +<br /> +<pre> +"result":[ + {"metric":{"instance":"192.168.2.130:9100","os":"freebsd"},"value":[...,"37.58"]}, + {"metric":{"instance":"192.168.2.131:9100","os":"freebsd"},"value":[...,"12.85"]}, + {"metric":{"instance":"192.168.2.132:9100","os":"freebsd"},"value":[...,"13.44"]} +] +</pre> +<br /> +<h3 style='display: inline' id='accessing-the-dashboards'>Accessing the Dashboards</h3><br /> +<br /> +<span>The dashboards are automatically imported by the Grafana sidecar and accessible at:</span><br /> +<br /> +<a class='textlink' href='https://grafana.f3s.buetow.org'>https://grafana.f3s.buetow.org</a><br /> +<br /> +<span>Navigate to Dashboards and search for:</span><br /> +<ul> +<li>"FreeBSD ZFS" - detailed per-host view with pool and dataset breakdowns</li> +<li>"FreeBSD ZFS Summary" - cluster-wide overview of all ZFS storage</li> +</ul><br /> +<h3 style='display: inline' id='key-metrics-to-monitor'>Key Metrics to Monitor</h3><br /> +<br /> +<span>**ARC Hit Rate:** Should typically be above 90% for optimal performance. Lower hit rates indicate the ARC cache is too small or workload has poor locality.</span><br /> +<br /> +<span>**ARC Memory Usage:** Shows how much of the maximum ARC size is being used. If consistently at or near maximum, the ARC is effectively utilizing available memory.</span><br /> +<br /> +<span>**Data vs Metadata:** Typically data should dominate, but workloads with many small files will show higher metadata percentages.</span><br /> +<br /> +<span>**MRU vs MFU:** Most Recently Used vs Most Frequently Used cache. The ratio depends on workload characteristics.</span><br /> +<br /> +<span>**Pool Capacity:** Monitor pool usage to ensure adequate free space. ZFS performance degrades when pools exceed 80% capacity.</span><br /> +<br /> +<span>**Pool Health:** Should always show ONLINE (green). DEGRADED (yellow) indicates a disk issue requiring attention. FAULTED (red) requires immediate action.</span><br /> +<br /> +<span>**Dataset Usage:** Track which datasets are consuming the most space to identify growth trends and plan capacity.</span><br /> +<br /> +<h3 style='display: inline' id='zfs-pool-and-dataset-metrics-via-textfile-collector'>ZFS Pool and Dataset Metrics via Textfile Collector</h3><br /> +<br /> +<span>To complement the ARC statistics from node_exporter's built-in ZFS collector, I added pool capacity and dataset metrics using the textfile collector feature.</span><br /> +<br /> +<span>Created a script at /usr/local/bin/zfs_pool_metrics.sh on each FreeBSD server:</span><br /> +<br /> +<pre> +#!/bin/sh +# ZFS Pool and Dataset Metrics Collector for Prometheus + +OUTPUT_FILE="/var/tmp/node_exporter/zfs_pools.prom.$$" +FINAL_FILE="/var/tmp/node_exporter/zfs_pools.prom" + +mkdir -p /var/tmp/node_exporter + +{ + # Pool metrics + echo "# HELP zfs_pool_size_bytes Total size of ZFS pool" + echo "# TYPE zfs_pool_size_bytes gauge" + echo "# HELP zfs_pool_allocated_bytes Allocated space in ZFS pool" + echo "# TYPE zfs_pool_allocated_bytes gauge" + echo "# HELP zfs_pool_free_bytes Free space in ZFS pool" + echo "# TYPE zfs_pool_free_bytes gauge" + echo "# HELP zfs_pool_capacity_percent Capacity percentage" + echo "# TYPE zfs_pool_capacity_percent gauge" + echo "# HELP zfs_pool_health Pool health (0=ONLINE, 1=DEGRADED, 2=FAULTED)" + echo "# TYPE zfs_pool_health gauge" + + zpool list -Hp -o name,size,allocated,free,capacity,health | \ + while IFS=$'\t' read name size alloc free cap health; do + case "$health" in + ONLINE) health_val=0 ;; + DEGRADED) health_val=1 ;; + FAULTED) health_val=2 ;; + *) health_val=6 ;; + esac + cap_num=$(echo "$cap" | sed 's/%//') + + echo "zfs_pool_size_bytes{pool=\"$name\"} $size" + echo "zfs_pool_allocated_bytes{pool=\"$name\"} $alloc" + echo "zfs_pool_free_bytes{pool=\"$name\"} $free" + echo "zfs_pool_capacity_percent{pool=\"$name\"} $cap_num" + echo "zfs_pool_health{pool=\"$name\"} $health_val" + done + + # Dataset metrics + echo "# HELP zfs_dataset_used_bytes Used space in dataset" + echo "# TYPE zfs_dataset_used_bytes gauge" + echo "# HELP zfs_dataset_available_bytes Available space" + echo "# TYPE zfs_dataset_available_bytes gauge" + echo "# HELP zfs_dataset_referenced_bytes Referenced space" + echo "# TYPE zfs_dataset_referenced_bytes gauge" + + zfs list -Hp -t filesystem -o name,used,available,referenced | \ + while IFS=$'\t' read name used avail ref; do + pool=$(echo "$name" | cut -d/ -f1) + echo "zfs_dataset_used_bytes{pool=\"$pool\",dataset=\"$name\"} $used" + echo "zfs_dataset_available_bytes{pool=\"$pool\",dataset=\"$name\"} $avail" + echo "zfs_dataset_referenced_bytes{pool=\"$pool\",dataset=\"$name\"} $ref" + done +} > "$OUTPUT_FILE" + +mv "$OUTPUT_FILE" "$FINAL_FILE" +</pre> +<br /> +<span>Deployed to all FreeBSD servers:</span><br /> +<br /> +<pre> +for host in f0 f1 f2; do + scp /tmp/zfs_pool_metrics.sh paul@$host:/tmp/ + ssh paul@$host 'doas mv /tmp/zfs_pool_metrics.sh /usr/local/bin/ && \ + doas chmod +x /usr/local/bin/zfs_pool_metrics.sh' +done +</pre> +<br /> +<span>Set up cron jobs to run every minute:</span><br /> +<br /> +<pre> +for host in f0 f1 f2; do + ssh paul@$host 'echo "* * * * * /usr/local/bin/zfs_pool_metrics.sh >/dev/null 2>&1" | \ + doas crontab -' +done +</pre> +<br /> +<span>The textfile collector (already configured with --collector.textfile.directory=/var/tmp/node_exporter) automatically picks up the metrics.</span><br /> +<br /> +<span>Verify metrics are being exposed:</span><br /> +<br /> +<pre> +paul@f0:~ % curl -s http://localhost:9100/metrics | grep "^zfs_pool" | head -5 +zfs_pool_allocated_bytes{pool="zdata"} 6.47622733824e+11 +zfs_pool_allocated_bytes{pool="zroot"} 5.3338578944e+10 +zfs_pool_capacity_percent{pool="zdata"} 64 +zfs_pool_capacity_percent{pool="zroot"} 10 +zfs_pool_free_bytes{pool="zdata"} 3.48809678848e+11 +</pre> +<br /> +<h2 style='display: inline' id='summary'>Summary</h2><br /> +<br /> +<span>Enhanced the f3s cluster observability by:</span><br /> +<br /> +<ul> +<li>Enabling etcd metrics monitoring for the k3s embedded etcd</li> +<li>Implementing comprehensive ZFS monitoring for FreeBSD storage servers</li> +<li>Creating recording rules for calculated metrics (ARC hit rates, memory usage, etc.)</li> +<li>Deploying Grafana dashboards for visualization</li> +<li>Configuring automatic dashboard import via ConfigMap labels</li> +</ul><br /> +<span>The monitoring stack now provides visibility into both cluster control plane health (etcd) and storage performance (ZFS).</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/prometheus'>prometheus configuration on Codeberg</a><br /> +<br /> +<h2 style='display: inline' id='distributed-tracing-with-grafana-tempo'>Distributed Tracing with Grafana Tempo</h2><br /> +<br /> +<span>After implementing logs (Loki) and metrics (Prometheus), the final pillar of observability is distributed tracing. Grafana Tempo provides distributed tracing capabilities that help understand request flows across microservices.</span><br /> +<br /> +<h3 style='display: inline' id='why-distributed-tracing'>Why Distributed Tracing?</h3><br /> +<br /> +<span>In a microservices architecture, a single user request may traverse multiple services. Distributed tracing:</span><br /> +<br /> +<ul> +<li>Tracks requests across service boundaries</li> +<li>Identifies performance bottlenecks</li> +<li>Visualizes service dependencies</li> +<li>Correlates with logs and metrics</li> +<li>Helps debug complex distributed systems</li> +</ul><br /> +<h3 style='display: inline' id='deploying-grafana-tempo'>Deploying Grafana Tempo</h3><br /> +<br /> +<span>Tempo is deployed in monolithic mode, following the same pattern as Loki's SingleBinary deployment.</span><br /> +<br /> +<span>#### Configuration Strategy</span><br /> +<br /> +<span>**Deployment Mode:** Monolithic (all components in one process)</span><br /> +<ul> +<li>Simpler operation than microservices mode</li> +<li>Suitable for the cluster scale</li> +<li>Consistent with Loki deployment pattern</li> +</ul><br /> +<span>**Storage:** Filesystem backend using hostPath</span><br /> +<ul> +<li>10Gi storage at /data/nfs/k3svolumes/tempo/data</li> +<li>7-day retention (168h)</li> +<li>Local storage is the only option for monolithic mode</li> +</ul><br /> +<span>**OTLP Receivers:** Standard OpenTelemetry Protocol ports</span><br /> +<ul> +<li>gRPC: 4317</li> +<li>HTTP: 4318</li> +<li>Bind to 0.0.0.0 to avoid Tempo 2.7+ localhost-only binding issue</li> +</ul><br /> +<span>#### Tempo Deployment Files</span><br /> +<br /> +<span>Created in /home/paul/git/conf/f3s/tempo/:</span><br /> +<br /> +<span>**values.yaml** - Helm chart configuration:</span><br /> +<br /> +<pre> +tempo: + retention: 168h + storage: + trace: + backend: local + local: + path: /var/tempo/traces + wal: + path: /var/tempo/wal + receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +persistence: + enabled: true + size: 10Gi + storageClassName: "" + +resources: + limits: + cpu: 1000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi +</pre> +<br /> +<span>**persistent-volumes.yaml** - Storage configuration:</span><br /> +<br /> +<pre> +apiVersion: v1 +kind: PersistentVolume +metadata: + name: tempo-data-pv +spec: + capacity: + storage: 10Gi + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + hostPath: + path: /data/nfs/k3svolumes/tempo/data +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: tempo-data-pvc + namespace: monitoring +spec: + storageClassName: "" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi +</pre> +<br /> +<span>**Grafana Datasource Provisioning**</span><br /> +<br /> +<span>All Grafana datasources (Prometheus, Alertmanager, Loki, Tempo) are provisioned via a unified ConfigMap that is directly mounted to the Grafana pod. This approach ensures datasources are loaded on startup without requiring sidecar-based discovery.</span><br /> +<br /> +<span>In /home/paul/git/conf/f3s/prometheus/grafana-datasources-all.yaml:</span><br /> +<br /> +<pre> +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-datasources-all + namespace: monitoring +data: + datasources.yaml: | + apiVersion: 1 + datasources: + - name: Prometheus + type: prometheus + uid: prometheus + url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/ + access: proxy + isDefault: true + - name: Alertmanager + type: alertmanager + uid: alertmanager + url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/ + - name: Loki + type: loki + uid: loki + url: http://loki.monitoring.svc.cluster.local:3100 + - name: Tempo + type: tempo + uid: tempo + url: http://tempo.monitoring.svc.cluster.local:3200 + jsonData: + tracesToLogsV2: + datasourceUid: loki + spanStartTimeShift: -1h + spanEndTimeShift: 1h + tracesToMetrics: + datasourceUid: prometheus + serviceMap: + datasourceUid: prometheus + nodeGraph: + enabled: true +</pre> +<br /> +<span>The kube-prometheus-stack Helm values (persistence-values.yaml) are configured to:</span><br /> +<ul> +<li>Disable sidecar-based datasource provisioning</li> +<li>Mount grafana-datasources-all ConfigMap directly to /etc/grafana/provisioning/datasources/</li> +</ul><br /> +<span>This direct mounting approach is simpler and more reliable than sidecar-based discovery.</span><br /> +<br /> +<span>#### Installation</span><br /> +<br /> +<pre> +cd /home/paul/git/conf/f3s/tempo +just install +</pre> +<br /> +<span>Verify Tempo is running:</span><br /> +<br /> +<pre> +kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo +kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready +</pre> +<br /> +<h3 style='display: inline' id='configuring-grafana-alloy-for-trace-collection'>Configuring Grafana Alloy for Trace Collection</h3><br /> +<br /> +<span>Updated /home/paul/git/conf/f3s/loki/alloy-values.yaml to add OTLP receivers for traces while maintaining existing log collection.</span><br /> +<br /> +<span>#### OTLP Receiver Configuration</span><br /> +<br /> +<span>Added to Alloy configuration after the log collection pipeline:</span><br /> +<br /> +<pre> +// OTLP receiver for traces via gRPC and HTTP +otelcol.receiver.otlp "default" { + grpc { + endpoint = "0.0.0.0:4317" + } + http { + endpoint = "0.0.0.0:4318" + } + output { + traces = [otelcol.processor.batch.default.input] + } +} + +// Batch processor for efficient trace forwarding +otelcol.processor.batch "default" { + timeout = "5s" + send_batch_size = 100 + send_batch_max_size = 200 + output { + traces = [otelcol.exporter.otlp.tempo.input] + } +} + +// OTLP exporter to send traces to Tempo +otelcol.exporter.otlp "tempo" { + client { + endpoint = "tempo.monitoring.svc.cluster.local:4317" + tls { + insecure = true + } + compression = "gzip" + } +} +</pre> +<br /> +<span>The batch processor reduces network overhead by accumulating spans before forwarding to Tempo.</span><br /> +<br /> +<span>#### Upgrade Alloy</span><br /> +<br /> +<pre> +cd /home/paul/git/conf/f3s/loki +just upgrade +</pre> +<br /> +<span>Verify OTLP receivers are listening:</span><br /> +<br /> +<pre> +kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "otlp.*receiver" +kubectl exec -n monitoring <alloy-pod> -- netstat -ln | grep -E ':(4317|4318)' +</pre> +<br /> +<h3 style='display: inline' id='demo-tracing-application'>Demo Tracing Application</h3><br /> +<br /> +<span>Created a three-tier Python application to demonstrate distributed tracing in action.</span><br /> +<br /> +<span>#### Application Architecture</span><br /> +<br /> +<pre> +User → Frontend (Flask:5000) → Middleware (Flask:5001) → Backend (Flask:5002) + ↓ ↓ ↓ + Alloy (OTLP:4317) → Tempo → Grafana +</pre> +<br /> +<span>**Frontend Service:**</span><br /> +<ul> +<li>Receives HTTP requests at /api/process</li> +<li>Forwards to middleware service</li> +<li>Creates parent span for the entire request</li> +</ul><br /> +<span>**Middleware Service:**</span><br /> +<ul> +<li>Transforms data at /api/transform</li> +<li>Calls backend service</li> +<li>Creates child span linked to frontend</li> +</ul><br /> +<span>**Backend Service:**</span><br /> +<ul> +<li>Returns data at /api/data</li> +<li>Simulates database query (100ms sleep)</li> +<li>Creates leaf span in the trace</li> +</ul><br /> +<span>#### OpenTelemetry Instrumentation</span><br /> +<br /> +<span>All services use Python OpenTelemetry libraries:</span><br /> +<br /> +<span>**Dependencies:**</span><br /> +<pre> +flask==3.0.0 +requests==2.31.0 +opentelemetry-distro==0.49b0 +opentelemetry-exporter-otlp==1.28.0 +opentelemetry-instrumentation-flask==0.49b0 +opentelemetry-instrumentation-requests==0.49b0 +</pre> +<br /> +<span>**Auto-instrumentation pattern** (used in all services):</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><b><u><font color="#000000">from</font></u></b> opentelemetry <b><u><font color="#000000">import</font></u></b> trace +<b><u><font color="#000000">from</font></u></b> opentelemetry.sdk.trace <b><u><font color="#000000">import</font></u></b> TracerProvider +<b><u><font color="#000000">from</font></u></b> opentelemetry.exporter.otlp.proto.grpc.trace_exporter <b><u><font color="#000000">import</font></u></b> OTLPSpanExporter +<b><u><font color="#000000">from</font></u></b> opentelemetry.instrumentation.flask <b><u><font color="#000000">import</font></u></b> FlaskInstrumentor +<b><u><font color="#000000">from</font></u></b> opentelemetry.instrumentation.requests <b><u><font color="#000000">import</font></u></b> RequestsInstrumentor +<b><u><font color="#000000">from</font></u></b> opentelemetry.sdk.resources <b><u><font color="#000000">import</font></u></b> Resource + +<i><font color="silver"># Define service identity</font></i> +resource = Resource(attributes={ + <font color="#808080">"service.name"</font>: <font color="#808080">"frontend"</font>, + <font color="#808080">"service.namespace"</font>: <font color="#808080">"tracing-demo"</font>, + <font color="#808080">"service.version"</font>: <font color="#808080">"1.0.0"</font> +}) + +provider = TracerProvider(resource=resource) + +<i><font color="silver"># Export to Alloy</font></i> +otlp_exporter = OTLPSpanExporter( + endpoint=<font color="#808080">"http://alloy.monitoring.svc.cluster.local:4317"</font>, + insecure=True +) + +processor = BatchSpanProcessor(otlp_exporter) +provider.add_span_processor(processor) +trace.set_tracer_provider(provider) + +<i><font color="silver"># Auto-instrument Flask and requests</font></i> +FlaskInstrumentor().instrument_app(app) +RequestsInstrumentor().instrument() +</pre> +<br /> +<span>The auto-instrumentation automatically:</span><br /> +<ul> +<li>Creates spans for HTTP requests</li> +<li>Propagates trace context via W3C Trace Context headers</li> +<li>Links parent and child spans across service boundaries</li> +</ul><br /> +<span>#### Deployment</span><br /> +<br /> +<span>Created Helm chart in /home/paul/git/conf/f3s/tracing-demo/ with three separate deployments, services, and an ingress.</span><br /> +<br /> +<span>Build and deploy:</span><br /> +<br /> +<pre> +cd /home/paul/git/conf/f3s/tracing-demo +just build +just import +just install +</pre> +<br /> +<span>Verify deployment:</span><br /> +<br /> +<pre> +kubectl get pods -n services | grep tracing-demo +kubectl get ingress -n services tracing-demo-ingress +</pre> +<br /> +<span>Access the application at:</span><br /> +<br /> +<a class='textlink' href='http://tracing-demo.f3s.buetow.org'>http://tracing-demo.f3s.buetow.org</a><br /> +<br /> +<h3 style='display: inline' id='visualizing-traces-in-grafana'>Visualizing Traces in Grafana</h3><br /> +<br /> +<span>The Tempo datasource is automatically discovered by Grafana through the ConfigMap label.</span><br /> +<br /> +<span>#### Accessing Traces</span><br /> +<br /> +<span>Navigate to Grafana → Explore → Select "Tempo" datasource</span><br /> +<br /> +<span>**Search Interface:**</span><br /> +<ul> +<li>Search by Trace ID</li> +<li>Search by service name</li> +<li>Search by tags</li> +</ul><br /> +<span>**TraceQL Queries:**</span><br /> +<br /> +<span>Find all traces from demo app:</span><br /> +<pre> +{ resource.service.namespace = "tracing-demo" } +</pre> +<br /> +<span>Find slow requests (>200ms):</span><br /> +<pre> +{ duration > 200ms } +</pre> +<br /> +<span>Find traces from specific service:</span><br /> +<pre> +{ resource.service.name = "frontend" } +</pre> +<br /> +<span>Find errors:</span><br /> +<pre> +{ status = error } +</pre> +<br /> +<span>Complex query - frontend traces calling middleware:</span><br /> +<pre> +{ resource.service.namespace = "tracing-demo" } && { span.http.status_code >= 500 } +</pre> +<br /> +<span>#### Service Graph Visualization</span><br /> +<br /> +<span>The service graph shows visual connections between services:</span><br /> +<br /> +<span>1. Navigate to Explore → Tempo</span><br /> +<span>2. Enable "Service Graph" view</span><br /> +<span>3. Shows: Frontend → Middleware → Backend with request rates</span><br /> +<br /> +<span>The service graph uses Prometheus metrics generated from trace data.</span><br /> +<br /> +<h3 style='display: inline' id='correlation-between-observability-signals'>Correlation Between Observability Signals</h3><br /> +<br /> +<span>Tempo integrates with Loki and Prometheus to provide unified observability.</span><br /> +<br /> +<span>#### Traces-to-Logs</span><br /> +<br /> +<span>Click on any span in a trace to see related logs:</span><br /> +<br /> +<span>1. View trace in Grafana</span><br /> +<span>2. Click on a span</span><br /> +<span>3. Select "Logs for this span"</span><br /> +<span>4. Loki shows logs filtered by:</span><br /> +<span> * Time range (span duration ± 1 hour)</span><br /> +<span> * Service name</span><br /> +<span> * Namespace</span><br /> +<span> * Pod</span><br /> +<br /> +<span>This helps correlate what the service was doing when the span was created.</span><br /> +<br /> +<span>#### Traces-to-Metrics</span><br /> +<br /> +<span>View Prometheus metrics for services in the trace:</span><br /> +<br /> +<span>1. View trace in Grafana</span><br /> +<span>2. Select "Metrics" tab</span><br /> +<span>3. Shows metrics like:</span><br /> +<span> * Request rate</span><br /> +<span> * Error rate</span><br /> +<span> * Duration percentiles</span><br /> +<br /> +<span>#### Logs-to-Traces</span><br /> +<br /> +<span>From logs, you can jump to related traces:</span><br /> +<br /> +<span>1. In Loki, logs that contain trace IDs are automatically linked</span><br /> +<span>2. Click the trace ID to view the full trace</span><br /> +<span>3. See the complete request flow</span><br /> +<br /> +<h3 style='display: inline' id='generating-traces-for-testing'>Generating Traces for Testing</h3><br /> +<br /> +<span>Test the demo application:</span><br /> +<br /> +<pre> +curl http://tracing-demo.f3s.buetow.org/api/process +</pre> +<br /> +<span>Load test (generates 50 traces):</span><br /> +<br /> +<pre> +cd /home/paul/git/conf/f3s/tracing-demo +just load-test +</pre> +<br /> +<span>Each request creates a distributed trace spanning all three services.</span><br /> +<br /> +<h3 style='display: inline' id='verifying-the-complete-pipeline'>Verifying the Complete Pipeline</h3><br /> +<br /> +<span>Check the trace flow end-to-end:</span><br /> +<br /> +<span>**1. Application generates traces:**</span><br /> +<pre> +kubectl logs -n services -l app=tracing-demo-frontend | grep -i trace +</pre> +<br /> +<span>**2. Alloy receives traces:**</span><br /> +<pre> +kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i otlp +</pre> +<br /> +<span>**3. Tempo stores traces:**</span><br /> +<pre> +kubectl logs -n monitoring -l app.kubernetes.io/name=tempo | grep -i trace +</pre> +<br /> +<span>**4. Grafana displays traces:**</span><br /> +<span>Navigate to Explore → Tempo → Search for traces</span><br /> +<br /> +<h3 style='display: inline' id='practical-example-viewing-a-distributed-trace'>Practical Example: Viewing a Distributed Trace</h3><br /> +<br /> +<span>Let's generate a trace and examine it in Grafana.</span><br /> +<br /> +<span>**1. Generate a trace by calling the demo application:**</span><br /> +<br /> +<pre> +curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process +</pre> +<br /> +<span>**Response (HTTP 200):**</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>{ + "middleware_response": { + "backend_data": { + "data": { + "id": <font color="#000000">12345</font>, + "query_time_ms": <font color="#000000">100.0</font>, + "timestamp": "<font color="#808080">2025-12-28T18:35:01.064538</font>", + "value": "<font color="#808080">Sample data from backend service</font>" + }, + "service": "<font color="#808080">backend</font>" + }, + "middleware_processed": <b><u><font color="#000000">true</font></u></b>, + "original_data": { + "source": "<font color="#808080">GET request</font>" + }, + "transformation_time_ms": <font color="#000000">50</font> + }, + "request_data": { + "source": "<font color="#808080">GET request</font>" + }, + "service": "<font color="#808080">frontend</font>", + "status": "<font color="#808080">success</font>" +} +</pre> +<br /> +<span>**2. Find the trace in Tempo via API:**</span><br /> +<br /> +<span>After a few seconds (for batch export), search for recent traces:</span><br /> +<br /> +<pre> +kubectl exec -n monitoring tempo-0 -- wget -qO- \ + 'http://localhost:3200/api/search?tags=service.namespace%3Dtracing-demo&limit=5' 2>/dev/null | \ + python3 -m json.tool +</pre> +<br /> +<span>Returns traces including:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>{ + "traceID": "<font color="#808080">4be1151c0bdcd5625ac7e02b98d95bd5</font>", + "rootServiceName": "<font color="#808080">frontend</font>", + "rootTraceName": "<font color="#808080">GET /api/process</font>", + "durationMs": <font color="#000000">221</font> +} +</pre> +<br /> +<span>**3. Fetch complete trace details:**</span><br /> +<br /> +<pre> +kubectl exec -n monitoring tempo-0 -- wget -qO- \ + 'http://localhost:3200/api/traces/4be1151c0bdcd5625ac7e02b98d95bd5' 2>/dev/null | \ + python3 -m json.tool +</pre> +<br /> +<span>**Trace structure (8 spans across 3 services):**</span><br /> +<br /> +<pre> +Trace ID: 4be1151c0bdcd5625ac7e02b98d95bd5 +Services: 3 (frontend, middleware, backend) + +Service: frontend + └─ GET /api/process 221.10ms (HTTP server span) + └─ frontend-process 216.23ms (custom business logic span) + └─ POST 209.97ms (HTTP client span to middleware) + +Service: middleware + └─ POST /api/transform 186.02ms (HTTP server span) + └─ middleware-transform 180.96ms (custom business logic span) + └─ GET 127.52ms (HTTP client span to backend) + +Service: backend + └─ GET /api/data 103.93ms (HTTP server span) + └─ backend-get-data 102.11ms (custom business logic span with 100ms sleep) +</pre> +<br /> +<span>**4. View the trace in Grafana UI:**</span><br /> +<br /> +<span>Navigate to: Grafana → Explore → Tempo datasource</span><br /> +<br /> +<span>Search using TraceQL:</span><br /> +<pre> +{ resource.service.namespace = "tracing-demo" } +</pre> +<br /> +<span>Or directly open the trace by pasting the trace ID in the search box:</span><br /> +<pre> +4be1151c0bdcd5625ac7e02b98d95bd5 +</pre> +<br /> +<span>**5. Trace visualization:**</span><br /> +<br /> +<span>The trace waterfall view in Grafana shows the complete request flow with timing:</span><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-8b/grafana-tempo-trace.png'><img alt='Distributed trace visualization in Grafana Tempo showing Frontend → Middleware → Backend spans' title='Distributed trace visualization in Grafana Tempo showing Frontend → Middleware → Backend spans' src='./f3s-kubernetes-with-freebsd-part-8b/grafana-tempo-trace.png' /></a><br /> +<br /> +<span>For additional examples of Tempo trace visualization, see also:</span><br /> +<br /> +<a class='textlink' href='https://foo.zone/gemfeed/2025-12-24-x-rag-observability-hackathon.html'>X-RAG Observability Hackathon (more Grafana Tempo screenshots)</a><br /> +<br /> +<span>The trace reveals the distributed request flow:</span><br /> +<ul> +<li>**Frontend (221ms)**: Receives GET /api/process, executes business logic, calls middleware</li> +<li>**Middleware (186ms)**: Receives POST /api/transform, transforms data, calls backend</li> +<li>**Backend (104ms)**: Receives GET /api/data, simulates database query with 100ms sleep</li> +<li>**Total request time**: 221ms end-to-end</li> +<li>**Span propagation**: W3C Trace Context headers automatically link all spans</li> +</ul><br /> +<span>**6. Service graph visualization:**</span><br /> +<br /> +<span>The service graph is automatically generated from traces and shows service dependencies. For examples of service graph visualization in Grafana, see the screenshots in the X-RAG Observability Hackathon blog post.</span><br /> +<br /> +<a class='textlink' href='https://foo.zone/gemfeed/2025-12-24-x-rag-observability-hackathon.html'>X-RAG Observability Hackathon (includes service graph screenshots)</a><br /> +<br /> +<span>This visualization helps identify:</span><br /> +<ul> +<li>Request rates between services</li> +<li>Average latency for each hop</li> +<li>Error rates (if any)</li> +<li>Service dependencies and communication patterns</li> +</ul><br /> +<h3 style='display: inline' id='storage-and-retention'>Storage and Retention</h3><br /> +<br /> +<span>Monitor Tempo storage usage:</span><br /> +<br /> +<pre> +kubectl exec -n monitoring <tempo-pod> -- df -h /var/tempo +</pre> +<br /> +<span>With 10Gi storage and 7-day retention, the system handles moderate trace volumes. If storage fills up:</span><br /> +<br /> +<ul> +<li>Reduce retention to 72h (3 days)</li> +<li>Implement sampling in Alloy</li> +<li>Increase PV size</li> +</ul><br /> +<h3 style='display: inline' id='complete-observability-stack'>Complete Observability Stack</h3><br /> +<br /> +<span>The f3s cluster now has complete observability:</span><br /> +<br /> +<span>**Metrics** (Prometheus):</span><br /> +<ul> +<li>Cluster resource usage</li> +<li>Application metrics</li> +<li>Node metrics (FreeBSD ZFS, OpenBSD edge)</li> +<li>etcd health</li> +</ul><br /> +<span>**Logs** (Loki):</span><br /> +<ul> +<li>All pod logs</li> +<li>Structured log collection</li> +<li>Log aggregation and search</li> +</ul><br /> +<span>**Traces** (Tempo):</span><br /> +<ul> +<li>Distributed request tracing</li> +<li>Service dependency mapping</li> +<li>Performance profiling</li> +<li>Error tracking</li> +</ul><br /> +<span>**Visualization** (Grafana):</span><br /> +<ul> +<li>Unified dashboards</li> +<li>Correlation between metrics, logs, and traces</li> +<li>Service graphs</li> +<li>Alerts</li> +</ul><br /> +<h3 style='display: inline' id='configuration-files'>Configuration Files</h3><br /> +<br /> +<span>All configuration files are available on Codeberg:</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/tempo'>Tempo configuration</a><br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/loki'>Alloy configuration (updated for traces)</a><br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/tracing-demo'>Demo tracing application</a><br /> +<p class="footer"> + Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> | + served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | + <a href="https://foo.zone/site-mirrors.html">Site Mirrors</a> + <br /> + Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a> +</p> +</body> +</html> diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 7e7f3733..09efca2f 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> - <updated>2025-12-26T23:33:35+02:00</updated> + <updated>2025-12-30T10:15:58+02:00</updated> <title>foo.zone feed</title> <subtitle>To be in the .zone!</subtitle> <link href="https://foo.zone/gemfeed/atom.xml" rel="self" /> @@ -2579,7 +2579,7 @@ p hash.values_at(:a, :c) <title>f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</title> <link href="https://foo.zone/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html" /> <id>https://foo.zone/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html</id> - <updated>2025-10-02T11:27:19+03:00</updated> + <updated>2025-10-02T11:27:19+03:00, last updated Tue 30 Dec 10:11:58 EET 2025</updated> <author> <name>Paul Buetow aka snonux</name> <email>paul@dev.buetow.org</email> @@ -2589,7 +2589,7 @@ p hash.values_at(:a, :c) <div xmlns="http://www.w3.org/1999/xhtml"> <h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-7-k3s-and-first-pod-deployments'>f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</h1><br /> <br /> -<span class='quote'>Published at 2025-10-02T11:27:19+03:00</span><br /> +<span class='quote'>Published at 2025-10-02T11:27:19+03:00, last updated Tue 30 Dec 10:11:58 EET 2025</span><br /> <br /> <span>This is the seventh blog post about the f3s series for my self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.</span><br /> <br /> @@ -2619,6 +2619,8 @@ p hash.values_at(:a, :c) <li>⇢ ⇢ <a href='#scaling-traefik-for-faster-failover'>Scaling Traefik for faster failover</a></li> <li>⇢ <a href='#make-it-accessible-from-the-public-internet'>Make it accessible from the public internet</a></li> <li>⇢ ⇢ <a href='#openbsd-relayd-configuration'>OpenBSD relayd configuration</a></li> +<li>⇢ ⇢ <a href='#automatic-failover-when-f3s-cluster-is-down'>Automatic failover when f3s cluster is down</a></li> +<li>⇢ ⇢ <a href='#openbsd-httpd-fallback-configuration'>OpenBSD httpd fallback configuration</a></li> <li>⇢ <a href='#deploying-the-private-docker-image-registry'>Deploying the private Docker image registry</a></li> <li>⇢ ⇢ <a href='#prepare-the-nfs-backed-storage'>Prepare the NFS-backed storage</a></li> <li>⇢ ⇢ <a href='#install-or-upgrade-the-chart'>Install (or upgrade) the chart</a></li> @@ -3248,10 +3250,11 @@ table <f3s> { } </pre> <br /> -<span>Inside the <span class='inlinecode'>http protocol "https"</span> block each public hostname gets its Let's Encrypt certificate and is matched to that backend table. Besides the primary trio, every service-specific hostname (<span class='inlinecode'>anki</span>, <span class='inlinecode'>bag</span>, <span class='inlinecode'>flux</span>, <span class='inlinecode'>audiobookshelf</span>, <span class='inlinecode'>gpodder</span>, <span class='inlinecode'>radicale</span>, <span class='inlinecode'>vault</span>, <span class='inlinecode'>syncthing</span>, <span class='inlinecode'>uprecords</span>) and their <span class='inlinecode'>www</span> / <span class='inlinecode'>standby</span> aliases reuse the same pool so new apps can go live just by publishing an ingress rule, whereas they will all map to a service running in k3s:</span><br /> +<span>Inside the <span class='inlinecode'>http protocol "https"</span> block each public hostname gets its Let's Encrypt certificate. The protocol configures TLS keypairs for all f3s services and other public endpoints. For f3s hosts specifically, there are no explicit <span class='inlinecode'>forward to</span> rules in the protocol—they use the relay-level failover mechanism described later. Non-f3s hosts get explicit localhost routing to prevent them from trying the f3s backends:</span><br /> <br /> <pre> http protocol "https" { + # TLS certificates for all f3s services tls keypair f3s.foo.zone tls keypair www.f3s.foo.zone tls keypair standby.f3s.foo.zone @@ -3283,36 +3286,15 @@ http protocol "https" { tls keypair www.uprecords.f3s.foo.zone tls keypair standby.uprecords.f3s.foo.zone - match request quick header "Host" value "f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "anki.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.anki.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.anki.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "bag.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.bag.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.bag.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "flux.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.flux.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.flux.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "audiobookshelf.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.audiobookshelf.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.audiobookshelf.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "gpodder.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.gpodder.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.gpodder.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "radicale.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.radicale.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.radicale.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "vault.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.vault.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.vault.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "syncthing.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.syncthing.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.syncthing.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "uprecords.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "www.uprecords.f3s.foo.zone" forward to <f3s> - match request quick header "Host" value "standby.uprecords.f3s.foo.zone" forward to <f3s> + # Explicitly route non-f3s hosts to localhost + match request header "Host" value "foo.zone" forward to <localhost> + match request header "Host" value "www.foo.zone" forward to <localhost> + match request header "Host" value "dtail.dev" forward to <localhost> + # ... other non-f3s hosts ... + + # NOTE: f3s hosts have NO match rules here! + # They use relay-level failover (f3s -> localhost backup) + # See the relay configuration below for automatic failover details } </pre> <br /> @@ -3322,18 +3304,143 @@ http protocol "https" { relay "https4" { listen on 46.23.94.99 port 443 tls protocol "https" + # Primary: f3s cluster (with health checks) - Falls back to localhost when all hosts down forward to <f3s> port 80 check tcp + forward to <localhost> port 8080 } relay "https6" { listen on 2a03:6000:6f67:624::99 port 443 tls protocol "https" + # Primary: f3s cluster (with health checks) - Falls back to localhost when all hosts down forward to <f3s> port 80 check tcp + forward to <localhost> port 8080 } </pre> <br /> <span>In practice, that means relayd terminates TLS with the correct certificate, keeps the three WireGuard-connected backends in rotation, and ships each request to whichever bhyve VM answers first.</span><br /> <br /> +<h3 style='display: inline' id='automatic-failover-when-f3s-cluster-is-down'>Automatic failover when f3s cluster is down</h3><br /> +<br /> +<span class='quote'>Update: This section was added at Tue 30 Dec 10:11:44 EET 2025</span><br /> +<br /> +<span>One important aspect of this setup is graceful degradation: when all three f3s nodes are unreachable (e.g., during maintenance or a power outage in my LAN), users should see a friendly status page instead of an error message.</span><br /> +<br /> +<span>OpenBSD's relayd supports automatic failover through its health check mechanism. According to the relayd.conf manual:</span><br /> +<br /> +<span class='quote'>This directive can be specified multiple times - subsequent entries will be used as the backup table if all hosts in the previous table are down.</span><br /> +<br /> +<span>The key is the order of <span class='inlinecode'>forward to</span> statements in the relay configuration. By placing the f3s table first with <span class='inlinecode'>check tcp</span> health checks, followed by localhost as a backup, relayd automatically routes traffic based on backend availability:</span><br /> +<br /> +<span>When f3s cluster is UP:</span><br /> +<br /> +<ul> +<li>Health checks on port 80 succeed for f3s nodes</li> +<li>All f3s traffic routes to the Kubernetes cluster</li> +<li>Localhost backup remains idle</li> +</ul><br /> +<span>When f3s cluster is DOWN:</span><br /> +<br /> +<ul> +<li>All health checks fail (nodes unreachable)</li> +<li>The <span class='inlinecode'><f3s></span> table becomes unavailable</li> +<li>Traffic automatically falls back to <span class='inlinecode'><localhost></span> on port 8080</li> +<li>OpenBSD's httpd serves a static fallback page</li> +</ul><br /> +<pre> +# NEW configuration - supports automatic failover +http protocol "https" { + # Explicitly route non-f3s hosts to localhost + match request header "Host" value "foo.zone" forward to <localhost> + match request header "Host" value "dtail.dev" forward to <localhost> + # ... other non-f3s hosts ... + + # f3s hosts have NO protocol rules - they use relay-level failover + # (no match rules for f3s.foo.zone, anki.f3s.foo.zone, etc.) +} + +relay "https4" { + # f3s FIRST (with health checks), localhost as BACKUP + forward to <f3s> port 80 check tcp + forward to <localhost> port 8080 +} +</pre> +<br /> +<span>This way, f3s traffic uses the relay's default behavior: try the first table, fall back to the second when health checks fail.</span><br /> +<br /> +<h3 style='display: inline' id='openbsd-httpd-fallback-configuration'>OpenBSD httpd fallback configuration</h3><br /> +<br /> +<span>The localhost httpd service on port 8080 serves the fallback content from <span class='inlinecode'>/var/www/htdocs/f3s_fallback/</span>. This directory contains a simple HTML page explaining the situation:</span><br /> +<br /> +<pre> +# OpenBSD httpd.conf +# Fallback for f3s hosts +server "f3s.foo.zone" { + listen on * port 8080 + log style forwarded + location * { + root "/htdocs/f3s_fallback" + directory auto index + } +} + +server "anki.f3s.foo.zone" { + listen on * port 8080 + log style forwarded + location * { + root "/htdocs/f3s_fallback" + directory auto index + } +} + +# ... similar blocks for all f3s hostnames ... +</pre> +<br /> +<span>The fallback page itself is straightforward:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><b><u><font color="#000000"><!DOCTYPE</font></u></b> <b><font color="#000000">html</font></b><b><u><font color="#000000">></font></u></b> +<b><u><font color="#000000"><html></font></u></b> +<b><u><font color="#000000"><head></font></u></b> + <b><u><font color="#000000"><title></font></u></b>Server turned off<b><u><font color="#000000"></title></font></u></b> + <b><u><font color="#000000"><style></font></u></b> + body { + font-family: <font color="#808080">sans-serif</font>; + text-align: <font color="#808080">center</font>; + padding-top: <font color="#808080">50px</font>; + } + .container { + max-width: <font color="#808080">600px</font>; + margin: <font color="#808080">0</font> <font color="#808080">auto</font>; + } + <b><u><font color="#000000"></style></font></u></b> +<b><u><font color="#000000"></head></font></u></b> +<b><u><font color="#000000"><body></font></u></b> + <b><u><font color="#000000"><div</font></u></b> <b><font color="#000000">class</font></b>=<font color="#808080">"container"</font><b><u><font color="#000000">></font></u></b> + <b><u><font color="#000000"><h1></font></u></b>Server turned off<b><u><font color="#000000"></h1></font></u></b> + <b><u><font color="#000000"><p></font></u></b>The servers are all currently turned off.<b><u><font color="#000000"></p></font></u></b> + <b><u><font color="#000000"><p></font></u></b>Please try again later.<b><u><font color="#000000"></p></font></u></b> + <b><u><font color="#000000"><p></font></u></b>Or email <b><u><font color="#000000"><a</font></u></b> <b><font color="#000000">href</font></b>=<font color="#808080">"mailto:paul@nospam.buetow.org"</font><b><u><font color="#000000">></font></u></b>paul@nospam.buetow.org<b><u><font color="#000000"></a></font></u></b> + - so I can turn them back on for you!<b><u><font color="#000000"></p></font></u></b> + <b><u><font color="#000000"></div></font></u></b> +<b><u><font color="#000000"></body></font></u></b> +<b><u><font color="#000000"></html></font></u></b> +</pre> +<br /> +<span>This approach provides several benefits:</span><br /> +<br /> +<ul> +<li>Automatic detection: Health checks run continuously; no manual intervention needed</li> +<li>Instant fallback: When all f3s nodes go down, the next request automatically routes to localhost</li> +<li>Transparent recovery: When f3s comes back online, health checks pass and traffic resumes automatically</li> +<li>User experience: Visitors see a helpful message instead of connection errors</li> +<li>No DNS changes: The same hostnames work whether f3s is up or down</li> +</ul><br /> +<span>This fallback mechanism has proven invaluable during maintenance windows and unexpected outages, ensuring that users always get a response even when the home lab is offline.</span><br /> +<br /> <h2 style='display: inline' id='deploying-the-private-docker-image-registry'>Deploying the private Docker image registry</h2><br /> <br /> <span>As not all Docker images I want to deploy are available on public Docker registries and as I also build some of them by myself, there is the need of a private registry. </span><br /> @@ -13,7 +13,7 @@ </p> <h1 style='display: inline' id='hello'>Hello!</h1><br /> <br /> -<span class='quote'>This site was generated at 2025-12-27T14:56:36+02:00 by <span class='inlinecode'>Gemtexter</span></span><br /> +<span class='quote'>This site was generated at 2025-12-30T10:15:58+02:00 by <span class='inlinecode'>Gemtexter</span></span><br /> <br /> <span>Welcome to the foo.zone!</span><br /> <br /> diff --git a/tags/style-override.css b/tags/style-override.css new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/tags/style-override.css diff --git a/test-tags.html.tmp b/test-tags.html.tmp new file mode 100644 index 00000000..cd05ebd2 --- /dev/null +++ b/test-tags.html.tmp @@ -0,0 +1,38 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<title>%%TITLE%%</title> +<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> +<link rel="stylesheet" href="%%STYLESHEET%%" /> +<link rel="stylesheet" href="%%STYLESHEET_OVERRIDE%%" /> +</head> +<body> +<p class="header"> +<a href="https://%%DOMAIN%%">Home</a> | <a href="%%MARKDOWN_BASE_URI%%%%CURRENT_PAGE%%.md">Markdown</a> | <a href="gemini://%%DOMAIN%%%%CURRENT_PAGE%%.gmi">Gemini</a> +</p> +<h1 style='display: inline' id='test-tags-page'>Test Tags Page</h1><br /> +<br /> +<span>This is a test page to demonstrate the tags feature.</span><br /> +<br /> +<span>This page is about <a class='textlink' href='../tags/bash.html'>#bash</a>, <a class='textlink' href='../tags/linux.html'>#linux</a>, and <a class='textlink' href='../tags/programming.html'>#programming</a>.</span><br /> +<br /> +<span>We also talk about <a class='textlink' href='../tags/golang.html'>#golang</a> and <a class='textlink' href='../tags/devops.html'>#devops</a> here.</span><br /> +<br /> +<h2 style='display: inline' id='code-example'>Code Example</h2><br /> +<br /> +<pre> +# This #tag-in-code-block should be ignored +echo "Hello World" +</pre> +<br /> +<span>Back to regular content with <a class='textlink' href='../tags/testing.html'>#testing</a> tag.</span><br /> +<p class="footer"> + Generated with <a href="https://codeberg.org/snonux/gemtexter">%%GEMTEXTER%%</a> | + served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | + <a href="https://%%DOMAIN%%/site-mirrors.html">Site Mirrors</a> + <br /> + Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a> +</p> +</body> +</html> diff --git a/test-template-tags.html.tmp b/test-template-tags.html.tmp new file mode 100644 index 00000000..fec41ad4 --- /dev/null +++ b/test-template-tags.html.tmp @@ -0,0 +1,38 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<title>%%TITLE%%</title> +<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> +<link rel="stylesheet" href="%%STYLESHEET%%" /> +<link rel="stylesheet" href="%%STYLESHEET_OVERRIDE%%" /> +</head> +<body> +<p class="header"> +<a href="https://%%DOMAIN%%">Home</a> | <a href="%%MARKDOWN_BASE_URI%%%%CURRENT_PAGE%%.md">Markdown</a> | <a href="gemini://%%DOMAIN%%%%CURRENT_PAGE%%.gmi">Gemini</a> +</p> +<h1 style='display: inline' id='template-tags-test'>Template Tags Test</h1><br /> +<br /> +<span>This file demonstrates the template::inline::tags feature.</span><br /> +<br /> +<br /> +<h2 style='display: inline' id='tags'>Tags</h2><br /> +<br /> +<a class='textlink' href='../tags/golang.html'><a class='textlink' href='../tags/golang.html'>#golang</a></a><br /> +<a class='textlink' href='../tags/programming.html'><a class='textlink' href='../tags/programming.html'>#programming</a></a><br /> +<a class='textlink' href='../tags/testing.html'><a class='textlink' href='../tags/testing.html'>#testing</a></a><br /> +<br /> +<span>This page is about <a class='textlink' href='../tags/golang.html'>#golang</a> and <a class='textlink' href='../tags/testing.html'>#testing</a>.</span><br /> +<br /> +<h2 style='display: inline' id='content'>Content</h2><br /> +<br /> +<span>Some content here about <a class='textlink' href='../tags/programming.html'>#programming</a>.</span><br /> +<p class="footer"> + Generated with <a href="https://codeberg.org/snonux/gemtexter">%%GEMTEXTER%%</a> | + served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | + <a href="https://%%DOMAIN%%/site-mirrors.html">Site Mirrors</a> + <br /> + Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a> +</p> +</body> +</html> diff --git a/uptime-stats.html b/uptime-stats.html index 693fb928..e46abc74 100644 --- a/uptime-stats.html +++ b/uptime-stats.html @@ -13,7 +13,7 @@ </p> <h1 style='display: inline' id='my-machine-uptime-stats'>My machine uptime stats</h1><br /> <br /> -<span class='quote'>This site was last updated at 2025-12-27T14:56:36+02:00</span><br /> +<span class='quote'>This site was last updated at 2025-12-30T10:15:58+02:00</span><br /> <br /> <span>The following stats were collected via <span class='inlinecode'>uptimed</span> on all of my personal computers over many years and the output was generated by <span class='inlinecode'>guprecords</span>, the global uptime records stats analyser of mine.</span><br /> <br /> @@ -35,24 +35,24 @@ | Pos | Host | Boots | Last Kernel | +-----+----------------+-------+-------------------------------+ | 1. | alphacentauri | 671 | FreeBSD 11.4-RELEASE-p7 | -| 2. | *earth | 218 | Linux 6.17.12-300.fc43.x86_64 | +| 2. | *earth | 220 | Linux 6.17.12-300.fc43.x86_64 | | 3. | mars | 207 | Linux 3.2.0-4-amd64 | | 4. | callisto | 153 | Linux 4.0.4-303.fc22.x86_64 | | 5. | dionysus | 136 | FreeBSD 13.0-RELEASE-p11 | | 6. | tauceti-e | 120 | Linux 3.2.0-4-amd64 | | 7. | *makemake | 81 | Linux 6.9.9-200.fc40.x86_64 | -| 8. | *f2 | 70 | FreeBSD 14.3-RELEASE | -| 9. | *f1 | 65 | FreeBSD 14.3-RELEASE | -| 10. | *f0 | 62 | FreeBSD 14.3-RELEASE | +| 8. | f2 | 70 | FreeBSD 14.3-RELEASE | +| 9. | f1 | 65 | FreeBSD 14.3-RELEASE | +| 10. | f0 | 62 | FreeBSD 14.3-RELEASE | | 11. | uranus | 59 | NetBSD 10.1 | | 12. | pluto | 51 | Linux 3.2.0-4-amd64 | -| 13. | *mega-m3-pro | 50 | Darwin 24.6.0 | -| 14. | mega15289 | 50 | Darwin 23.4.0 | -| 15. | *fishfinger | 46 | OpenBSD 7.7 | +| 13. | mega15289 | 50 | Darwin 23.4.0 | +| 14. | *mega-m3-pro | 50 | Darwin 24.6.0 | +| 15. | fishfinger | 46 | OpenBSD 7.7 | | 16. | t450 | 44 | FreeBSD 14.2-RELEASE | -| 17. | *blowfish | 43 | OpenBSD 7.7 | -| 18. | phobos | 40 | Linux 3.4.0-CM-g1dd7cdf | -| 19. | mega8477 | 40 | Darwin 13.4.0 | +| 17. | blowfish | 43 | OpenBSD 7.7 | +| 18. | mega8477 | 40 | Darwin 13.4.0 | +| 19. | phobos | 40 | Linux 3.4.0-CM-g1dd7cdf | | 20. | sun | 33 | FreeBSD 10.3-RELEASE-p24 | +-----+----------------+-------+-------------------------------+ </pre> @@ -66,12 +66,12 @@ | Pos | Host | Uptime | Last Kernel | +-----+----------------+-----------------------------+-----------------------------------+ | 1. | vulcan | 4 years, 5 months, 6 days | Linux 3.10.0-1160.81.1.el7.x86_64 | -| 2. | *earth | 3 years, 12 months, 3 days | Linux 6.17.12-300.fc43.x86_64 | -| 3. | *blowfish | 3 years, 10 months, 2 days | OpenBSD 7.7 | +| 2. | *earth | 3 years, 12 months, 6 days | Linux 6.17.12-300.fc43.x86_64 | +| 3. | blowfish | 3 years, 10 months, 2 days | OpenBSD 7.7 | | 4. | sun | 3 years, 9 months, 26 days | FreeBSD 10.3-RELEASE-p24 | | 5. | uranus | 3 years, 9 months, 5 days | NetBSD 10.1 | | 6. | uugrn | 3 years, 5 months, 5 days | FreeBSD 11.2-RELEASE-p4 | -| 7. | *fishfinger | 3 years, 1 months, 28 days | OpenBSD 7.7 | +| 7. | fishfinger | 3 years, 1 months, 28 days | OpenBSD 7.7 | | 8. | deltavega | 3 years, 1 months, 21 days | Linux 3.10.0-1160.11.1.el7.x86_64 | | 9. | pluto | 2 years, 10 months, 29 days | Linux 3.2.0-4-amd64 | | 10. | tauceti | 2 years, 3 months, 19 days | Linux 3.2.0-4-amd64 | @@ -97,13 +97,13 @@ | Pos | Host | Score | Last Kernel | +-----+----------------+-------+-----------------------------------+ | 1. | uranus | 340 | NetBSD 10.1 | -| 2. | vulcan | 275 | Linux 3.10.0-1160.81.1.el7.x86_64 | -| 3. | *earth | 275 | Linux 6.17.12-300.fc43.x86_64 | -| 4. | *blowfish | 243 | OpenBSD 7.7 | +| 2. | *earth | 276 | Linux 6.17.12-300.fc43.x86_64 | +| 3. | vulcan | 275 | Linux 3.10.0-1160.81.1.el7.x86_64 | +| 4. | blowfish | 240 | OpenBSD 7.7 | | 5. | sun | 238 | FreeBSD 10.3-RELEASE-p24 | | 6. | uugrn | 211 | FreeBSD 11.2-RELEASE-p4 | | 7. | alphacentauri | 201 | FreeBSD 11.4-RELEASE-p7 | -| 8. | *fishfinger | 200 | OpenBSD 7.7 | +| 8. | fishfinger | 198 | OpenBSD 7.7 | | 9. | deltavega | 193 | Linux 3.10.0-1160.11.1.el7.x86_64 | | 10. | pluto | 182 | Linux 3.2.0-4-amd64 | | 11. | dionysus | 156 | FreeBSD 13.0-RELEASE-p11 | @@ -138,9 +138,9 @@ | 9. | mars | 1 years, 2 months, 10 days | Linux 3.2.0-4-amd64 | | 10. | tauceti-e | 0 years, 12 months, 9 days | Linux 3.2.0-4-amd64 | | 11. | sirius | 0 years, 8 months, 20 days | Linux 2.6.32-042stab111.12 | -| 12. | *f0 | 0 years, 8 months, 3 days | FreeBSD 14.3-RELEASE | -| 13. | *f2 | 0 years, 8 months, 2 days | FreeBSD 14.3-RELEASE | -| 14. | *f1 | 0 years, 8 months, 1 days | FreeBSD 14.3-RELEASE | +| 12. | f0 | 0 years, 8 months, 3 days | FreeBSD 14.3-RELEASE | +| 13. | f2 | 0 years, 8 months, 2 days | FreeBSD 14.3-RELEASE | +| 14. | f1 | 0 years, 8 months, 1 days | FreeBSD 14.3-RELEASE | | 15. | *earth | 0 years, 6 months, 29 days | Linux 6.17.12-300.fc43.x86_64 | | 16. | deimos | 0 years, 5 months, 15 days | Linux 4.4.5-300.fc23.x86_64 | | 17. | joghurt | 0 years, 2 months, 9 days | FreeBSD 7.0-PRERELEASE | @@ -162,13 +162,13 @@ | 2. | dionysus | 8 years, 6 months, 17 days | FreeBSD 13.0-RELEASE-p11 | | 3. | alphacentauri | 6 years, 9 months, 13 days | FreeBSD 11.4-RELEASE-p7 | | 4. | *makemake | 4 years, 10 months, 16 days | Linux 6.9.9-200.fc40.x86_64 | -| 5. | *earth | 4 years, 6 months, 1 days | Linux 6.17.12-300.fc43.x86_64 | +| 5. | *earth | 4 years, 6 months, 4 days | Linux 6.17.12-300.fc43.x86_64 | | 6. | vulcan | 4 years, 5 months, 6 days | Linux 3.10.0-1160.81.1.el7.x86_64 | -| 7. | *blowfish | 3 years, 10 months, 3 days | OpenBSD 7.7 | +| 7. | blowfish | 3 years, 10 months, 3 days | OpenBSD 7.7 | | 8. | sun | 3 years, 10 months, 2 days | FreeBSD 10.3-RELEASE-p24 | | 9. | uugrn | 3 years, 5 months, 5 days | FreeBSD 11.2-RELEASE-p4 | | 10. | mega15289 | 3 years, 4 months, 9 days | Darwin 23.4.0 | -| 11. | *fishfinger | 3 years, 1 months, 30 days | OpenBSD 7.7 | +| 11. | fishfinger | 3 years, 1 months, 30 days | OpenBSD 7.7 | | 12. | deltavega | 3 years, 1 months, 21 days | Linux 3.10.0-1160.11.1.el7.x86_64 | | 13. | pluto | 2 years, 10 months, 30 days | Linux 3.2.0-4-amd64 | | 14. | t450 | 2 years, 9 months, 6 days | FreeBSD 14.2-RELEASE | @@ -191,13 +191,13 @@ +-----+----------------+-------+ | 1. | FreeBSD 10... | 551 | | 2. | Linux 3... | 550 | -| 3. | *FreeBSD 14... | 215 | -| 4. | *Linux 6... | 203 | +| 3. | FreeBSD 14... | 215 | +| 4. | *Linux 6... | 205 | | 5. | Linux 5... | 162 | | 6. | Linux 4... | 161 | | 7. | FreeBSD 11... | 153 | | 8. | FreeBSD 13... | 116 | -| 9. | *OpenBSD 7... | 99 | +| 9. | OpenBSD 7... | 99 | | 10. | Darwin 13... | 40 | | 11. | Darwin 23... | 29 | | 12. | *Darwin 24... | 26 | @@ -207,8 +207,8 @@ | 16. | Darwin 15... | 15 | | 17. | Darwin 22... | 12 | | 18. | Darwin 18... | 11 | -| 19. | OpenBSD 4... | 10 | -| 20. | FreeBSD 7... | 10 | +| 19. | FreeBSD 6... | 10 | +| 20. | OpenBSD 4... | 10 | +-----+----------------+-------+ </pre> <br /> @@ -221,13 +221,13 @@ | Pos | KernelMajor | Uptime | +-----+----------------+------------------------------+ | 1. | Linux 3... | 15 years, 10 months, 25 days | -| 2. | *OpenBSD 7... | 7 years, 6 months, 29 days | +| 2. | OpenBSD 7... | 7 years, 6 months, 29 days | | 3. | FreeBSD 10... | 5 years, 9 months, 9 days | | 4. | Linux 5... | 4 years, 10 months, 21 days | -| 5. | *Linux 6... | 3 years, 3 months, 3 days | +| 5. | *Linux 6... | 3 years, 3 months, 7 days | | 6. | Linux 4... | 2 years, 7 months, 22 days | | 7. | FreeBSD 11... | 2 years, 4 months, 28 days | -| 8. | *FreeBSD 14... | 2 years, 3 months, 24 days | +| 8. | FreeBSD 14... | 2 years, 3 months, 24 days | | 9. | Linux 2... | 1 years, 11 months, 21 days | | 10. | Darwin 13... | 1 years, 3 months, 25 days | | 11. | FreeBSD 6... | 1 years, 3 months, 9 days | @@ -252,12 +252,12 @@ | Pos | KernelMajor | Score | +-----+----------------+-------+ | 1. | Linux 3... | 1045 | -| 2. | *OpenBSD 7... | 484 | +| 2. | OpenBSD 7... | 481 | | 3. | FreeBSD 10... | 406 | | 4. | Linux 5... | 317 | | 5. | *Linux 6... | 220 | | 6. | Linux 4... | 175 | -| 7. | *FreeBSD 14... | 161 | +| 7. | FreeBSD 14... | 159 | | 8. | FreeBSD 11... | 159 | | 9. | Linux 2... | 121 | | 10. | Darwin 13... | 80 | @@ -269,8 +269,8 @@ | 16. | Darwin 18... | 32 | | 17. | Darwin 22... | 30 | | 18. | Darwin 15... | 29 | -| 19. | FreeBSD 5... | 25 | -| 20. | FreeBSD 13... | 25 | +| 19. | FreeBSD 13... | 25 | +| 20. | FreeBSD 5... | 25 | +-----+----------------+-------+ </pre> <br /> @@ -282,10 +282,10 @@ +-----+------------+-------+ | Pos | KernelName | Boots | +-----+------------+-------+ -| 1. | *Linux | 1098 | -| 2. | *FreeBSD | 1080 | +| 1. | *Linux | 1100 | +| 2. | FreeBSD | 1080 | | 3. | *Darwin | 155 | -| 4. | *OpenBSD | 109 | +| 4. | OpenBSD | 109 | | 5. | NetBSD | 1 | +-----+------------+-------+ </pre> @@ -298,9 +298,9 @@ +-----+------------+-----------------------------+ | Pos | KernelName | Uptime | +-----+------------+-----------------------------+ -| 1. | *Linux | 28 years, 3 months, 26 days | -| 2. | *FreeBSD | 12 years, 2 months, 24 days | -| 3. | *OpenBSD | 8 years, 2 months, 7 days | +| 1. | *Linux | 28 years, 3 months, 30 days | +| 2. | FreeBSD | 12 years, 2 months, 24 days | +| 3. | OpenBSD | 8 years, 2 months, 7 days | | 4. | *Darwin | 5 years, 2 months, 22 days | | 5. | NetBSD | 0 years, 1 months, 1 days | +-----+------------+-----------------------------+ @@ -314,9 +314,9 @@ +-----+------------+-------+ | Pos | KernelName | Score | +-----+------------+-------+ -| 1. | *Linux | 1879 | -| 2. | *FreeBSD | 862 | -| 3. | *OpenBSD | 523 | +| 1. | *Linux | 1880 | +| 2. | FreeBSD | 860 | +| 3. | OpenBSD | 520 | | 4. | *Darwin | 340 | | 5. | NetBSD | 0 | +-----+------------+-------+ |