Update content for html

7 files changed, 1435 insertions, 168 deletions

diff --git a/about/resources.html b/about/resources.html
index 3bada362..30c8db37 100644
--- a/about/resources.html
+++ b/about/resources.html
@@ -50,112 +50,112 @@
 <span>In random order:</span><br />
 <br />
 <ul>
-<li>Polished Ruby Programming; Jeremy Evans; Packt Publishing</li>
+<li>DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible</li>
+<li>100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications</li>
+<li>Raku Recipes; J.J. Merelo; Apress</li>
+<li>Perl New Features; Joshua McAdams, brian d foy; Perl School</li>
+<li>97 things every SRE should know; Emil Stolarsky, Jaime Woo; O&#39;Reilly</li>
+<li>The Pragmatic Programmer; David Thomas; Addison-Wesley</li>
+<li>The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional</li>
+<li>Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O&#39;Reilly</li>
+<li>21st Century C: C Tips from the New School; Ben Klemens; O&#39;Reilly</li>
+<li>The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton</li>
+<li>Terraform Cookbook; Mikael Krief; Packt Publishing</li>
+<li>The Docker Book; James Turnbull; Kindle</li>
 <li>Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook</li>
-<li>Concurrency in Go; Katherine Cox-Buday; O&#39;Reilly</li>
-<li>Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner</li>
-<li>Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson</li>
-<li>The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress</li>
+<li>Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook</li>
 <li>Effective Java; Joshua Bloch; Addison-Wesley Professional</li>
-<li>Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O&#39;Reilly</li>
-<li>Site Reliability Engineering; How Google runs production systems; O&#39;Reilly</li>
-<li>Data Science at the Command Line; Jeroen Janssens; O&#39;Reilly</li>
+<li>Clusterbau mit Linux-HA; Michael Schwartzkopff; O&#39;Reilly</li>
+<li>Effective awk programming; Arnold Robbins; O&#39;Reilly</li>
+<li>Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press</li>
+<li>The Kubernetes Book; Nigel Poulton; Unabridged Audiobook</li>
+<li>Funktionale Programmierung; Peter Pepper; Springer</li>
 <li>Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers </li>
-<li>DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible</li>
-<li>Ultimate Go Notebook; Bill Kennedy</li>
-<li>The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional</li>
-<li>DNS and BIND; Cricket Liu; O&#39;Reilly</li>
-<li>Modern Perl; Chromatic ; Onyx Neon Press</li>
-<li>Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O&#39;Reilly</li>
 <li>Pro Puppet; James Turnbull, Jeffrey McCune; Apress</li>
-<li>Raku Recipes; J.J. Merelo; Apress</li>
-<li>Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt </li>
-<li>Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press</li>
-<li>The Docker Book; James Turnbull; Kindle</li>
+<li>Polished Ruby Programming; Jeremy Evans; Packt Publishing</li>
 <li>Developing Games in Java; David Brackeen and others...; New Riders</li>
-<li>Higher Order Perl; Mark Dominus; Morgan Kaufmann</li>
-<li>97 things every SRE should know; Emil Stolarsky, Jaime Woo; O&#39;Reilly</li>
+<li>Ultimate Go Notebook; Bill Kennedy</li>
 <li>Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf</li>
-<li>Terraform Cookbook; Mikael Krief; Packt Publishing</li>
-<li>Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook</li>
 <li>Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers</li>
-<li>Leanring eBPF; Liz Rice; O&#39;Reilly</li>
+<li>Systemprogrammierung in Go; Frank Müller; dpunkt</li>
+<li>Data Science at the Command Line; Jeroen Janssens; O&#39;Reilly</li>
 <li>C++ Programming Language; Bjarne Stroustrup;</li>
-<li>Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O&#39;Reilly</li>
-<li>Java ist auch eine Insel; Christian Ullenboom; </li>
-<li>The Pragmatic Programmer; David Thomas; Addison-Wesley</li>
-<li>21st Century C: C Tips from the New School; Ben Klemens; O&#39;Reilly</li>
-<li>Funktionale Programmierung; Peter Pepper; Springer</li>
-<li>Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall &amp; Jon Orwant; O&#39;Reilly</li>
-<li>The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible</li>
+<li>Site Reliability Engineering; How Google runs production systems; O&#39;Reilly</li>
+<li>Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt </li>
 <li>Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press</li>
-<li>Clusterbau mit Linux-HA; Michael Schwartzkopff; O&#39;Reilly</li>
-<li>The Kubernetes Book; Nigel Poulton; Unabridged Audiobook</li>
-<li>100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications</li>
-<li>Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications</li>
-<li>Perl New Features; Joshua McAdams, brian d foy; Perl School</li>
-<li>The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton</li>
-<li>Systemprogrammierung in Go; Frank Müller; dpunkt</li>
-<li>Effective awk programming; Arnold Robbins; O&#39;Reilly</li>
+<li>Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner</li>
+<li>Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson</li>
 <li>Raku Fundamentals; Moritz Lenz; Apress</li>
+<li>Java ist auch eine Insel; Christian Ullenboom; </li>
+<li>Leanring eBPF; Liz Rice; O&#39;Reilly</li>
+<li>Modern Perl; Chromatic ; Onyx Neon Press</li>
+<li>Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications</li>
+<li>Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O&#39;Reilly</li>
+<li>The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible</li>
+<li>Higher Order Perl; Mark Dominus; Morgan Kaufmann</li>
+<li>Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall &amp; Jon Orwant; O&#39;Reilly</li>
+<li>The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress</li>
+<li>Concurrency in Go; Katherine Cox-Buday; O&#39;Reilly</li>
+<li>DNS and BIND; Cricket Liu; O&#39;Reilly</li>
+<li>Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O&#39;Reilly</li>
 </ul><br />
 <h2 style='display: inline' id='technical-references'>Technical references</h2><br />
 <br />
 <span>I didn&#39;t read them from the beginning to the end, but I am using them to look up things. The books are in random order:</span><br />
 <br />
 <ul>
-<li>Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O&#39;Reilly</li>
-<li>BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley</li>
-<li>Go: Design Patterns for Real-World Projects; Mat Ryer; Packt</li>
 <li>Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley</li>
 <li>Relayd and Httpd Mastery; Michael W Lucas</li>
+<li>BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley</li>
 <li>Groovy Kurz &amp; Gut; Joerg Staudemeier; O&#39;Reilly</li>
+<li>Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O&#39;Reilly</li>
 <li>The Linux Programming Interface; Michael Kerrisk; No Starch Press </li>
 <li>Implementing Service Level Objectives; Alex Hidalgo; O&#39;Reilly</li>
+<li>Go: Design Patterns for Real-World Projects; Mat Ryer; Packt</li>
 </ul><br />
 <h2 style='display: inline' id='self-development-and-soft-skills-books'>Self-development and soft-skills books</h2><br />
 <br />
 <span>In random order:</span><br />
 <br />
 <ul>
-<li>Ultralearning; Anna Laurent; Self-published via Amazon</li>
-<li>The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd</li>
-<li>Atomic Habits; James Clear; Random House Business</li>
-<li>Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook</li>
-<li>The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon &amp; Schuster UK</li>
+<li>Ultralearning; Scott Young; Thorsons</li>
+<li>Getting Things Done; David Allen</li>
 <li>Eat That Frog!; Brian Tracy; Hodder Paperbacks</li>
+<li>Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook</li>
+<li>Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne</li>
+<li>The Software Engineer&#39;s Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook </li>
+<li>Stop starting, start finishing; Arne Roock; Lean-Kanban University  </li>
+<li>Digital Minimalism; Cal Newport; Portofolio Penguin</li>
+<li>97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook</li>
+<li>Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook</li>
+<li>101 Essays that change the way you think; Brianna Wiest; Audiobook</li>
+<li>Soft Skills; John Sommez; Manning Publications</li>
+<li>Atomic Habits; James Clear; Random House Business</li>
+<li>Ultralearning; Anna Laurent; Self-published via Amazon</li>
+<li>The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME)</li>
 <li>The Good Enough Job; Simone Stolzoff; Ebury Edge</li>
-<li>Deep Work; Cal Newport; Piatkus</li>
+<li>Never Split the Difference; Chris Voss, Tahl Raz; Random House Business</li>
 <li>The Power of Now; Eckhard Tolle; Yellow Kite</li>
-<li>Psycho-Cybernetics; Maxwell Maltz; Perigee Books</li>
-<li>101 Essays that change the way you think; Brianna Wiest; Audiobook</li>
-<li>Digital Minimalism; Cal Newport; Portofolio Penguin</li>
-<li>The Software Engineer&#39;s Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook </li>
 <li>Influence without Authority; A. Cohen, D. Bradford; Wiley</li>
-<li>Never Split the Difference; Chris Voss, Tahl Raz; Random House Business</li>
-<li>The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME)</li>
-<li>Getting Things Done; David Allen</li>
-<li>Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook</li>
-<li>Eat That Frog; Brian Tracy</li>
-<li>The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook</li>
-<li>So Good They Can&#39;t Ignore You; Cal Newport; Business Plus</li>
+<li>Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press</li>
+<li>Meditation for Mortals, Oliver Burkeman, Audiobook</li>
+<li>Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion</li>
 <li>The Complete Software Developer&#39;s Career Guide; John Sonmez; Unabridged Audiobook</li>
+<li>The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon &amp; Schuster UK</li>
 <li>The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books</li>
-<li>Solve for Happy; Mo Gawdat (RE-READ 1ST TIME)</li>
-<li>Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne</li>
-<li>The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select</li>
-<li>Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion</li>
+<li>The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd</li>
+<li>Eat That Frog; Brian Tracy</li>
 <li>The Bullet Journal Method; Ryder Carroll; Fourth Estate</li>
-<li>Soft Skills; John Sommez; Manning Publications</li>
-<li>Meditation for Mortals, Oliver Burkeman, Audiobook</li>
-<li>Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press</li>
+<li>The Joy of Missing Out; Christina Crook; New Society Publishers</li>
+<li>So Good They Can&#39;t Ignore You; Cal Newport; Business Plus</li>
+<li>The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select</li>
 <li>Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing</li>
-<li>Stop starting, start finishing; Arne Roock; Lean-Kanban University  </li>
 <li>Time Management for System Administrators; Thomas A. Limoncelli; O&#39;Reilly</li>
-<li>The Joy of Missing Out; Christina Crook; New Society Publishers</li>
+<li>The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook</li>
+<li>Psycho-Cybernetics; Maxwell Maltz; Perigee Books</li>
+<li>Solve for Happy; Mo Gawdat (RE-READ 1ST TIME)</li>
+<li>Deep Work; Cal Newport; Piatkus</li>
 <li>Slow Productivity; Cal Newport; Penguin Random House</li>
-<li>Ultralearning; Scott Young; Thorsons</li>
-<li>97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook</li>
 </ul><br />
 <a class='textlink' href='../notes/index.html'>Here are notes of mine for some of the books</a><br />
 <br />
@@ -164,31 +164,31 @@
 <span>Some of these were in-person with exams; others were online learning lectures only. In random order:</span><br />
 <br />
 <ul>
-<li>Ultimate Go Programming; Bill Kennedy; O&#39;Reilly Online</li>
-<li>Structure and Interpretation of Computer Programs; Harold Abelson and more...; </li>
-<li>Apache Tomcat Best Practises; 3-day on-site training</li>
-<li>F5 Loadbalancers Training; 2-day on-site training; F5, Inc. </li>
-<li>Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon</li>
-<li>Protocol buffers; O&#39;Reilly Online</li>
-<li>Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training</li>
-<li>Algorithms Video Lectures; Robert Sedgewick; O&#39;Reilly Online</li>
 <li>Scripting Vim; Damian Conway; O&#39;Reilly Online</li>
-<li>AWS Immersion Day; Amazon; 1-day interactive online training </li>
+<li>Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon</li>
 <li>MySQL Deep Dive Workshop; 2-day on-site training</li>
+<li>F5 Loadbalancers Training; 2-day on-site training; F5, Inc. </li>
+<li>Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training</li>
 <li>Developing IaC with Terraform (with Live Lessons); O&#39;Reilly Online</li>
+<li>Protocol buffers; O&#39;Reilly Online</li>
+<li>The Well-Grounded Rubyist Video Edition; David. A. Black; O&#39;Reilly Online</li>
+<li>Functional programming lecture; Remote University of Hagen</li>
 <li>Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need)</li>
+<li>Structure and Interpretation of Computer Programs; Harold Abelson and more...; </li>
+<li>Ultimate Go Programming; Bill Kennedy; O&#39;Reilly Online</li>
+<li>AWS Immersion Day; Amazon; 1-day interactive online training </li>
+<li>Algorithms Video Lectures; Robert Sedgewick; O&#39;Reilly Online</li>
 <li>The Ultimate Kubernetes Bootcamp; School of Devops; O&#39;Reilly Online</li>
-<li>Functional programming lecture; Remote University of Hagen</li>
-<li>The Well-Grounded Rubyist Video Edition; David. A. Black; O&#39;Reilly Online</li>
+<li>Apache Tomcat Best Practises; 3-day on-site training</li>
 </ul><br />
 <h2 style='display: inline' id='technical-guides'>Technical guides</h2><br />
 <br />
 <span>These are not whole books, but guides (smaller or larger) which I found very useful. in random order:</span><br />
 <br />
 <ul>
+<li>How CPUs work at https://cpu.land</li>
 <li>Advanced Bash-Scripting Guide </li>
 <li>Raku Guide at https://raku.guide  </li>
-<li>How CPUs work at https://cpu.land</li>
 </ul><br />
 <h2 style='display: inline' id='podcasts'>Podcasts</h2><br />
 <br />
@@ -197,61 +197,61 @@
 <span>In random order:</span><br />
 <br />
 <ul>
-<li>The ProdCast (Google SRE Podcast)</li>
-<li>BSD Now [BSD]</li>
-<li>Pratical AI</li>
+<li>Wednesday Wisdom</li>
+<li>Dev Interrupted</li>
 <li>Deep Questions with Cal Newport</li>
+<li>Hidden Brain</li>
+<li>Cup o&#39; Go [Golang]</li>
 <li>Fork Around And Find Out</li>
-<li>The Changelog Podcast(s)</li>
-<li>Fallthrough [Golang]</li>
+<li>Pratical AI</li>
+<li>Modern Mentor</li>
 <li>Maintainable</li>
-<li>Cup o&#39; Go [Golang]</li>
-<li>Hidden Brain</li>
+<li>The Changelog Podcast(s)</li>
+<li>The ProdCast (Google SRE Podcast)</li>
+<li>BSD Now [BSD]</li>
 <li>Backend Banter</li>
+<li>Fallthrough [Golang]</li>
 <li>The Pragmatic Engineer Podcast</li>
-<li>Wednesday Wisdom</li>
-<li>Dev Interrupted</li>
-<li>Modern Mentor</li>
 </ul><br />
 <h3 style='display: inline' id='podcasts-i-liked'>Podcasts I liked</h3><br />
 <br />
 <span>I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests.</span><br />
 <br />
 <ul>
-<li>CRE: Chaosradio Express [german]</li>
-<li>Ship It (predecessor of Fork Around And Find Out)</li>
-<li>FLOSS weekly</li>
-<li>Modern Mentor</li>
 <li>Go Time (predecessor of fallthrough)</li>
+<li>Modern Mentor</li>
+<li>CRE: Chaosradio Express [german]</li>
 <li>Java Pub House</li>
+<li>FLOSS weekly</li>
+<li>Ship It (predecessor of Fork Around And Find Out)</li>
 </ul><br />
 <h2 style='display: inline' id='newsletters-i-like'>Newsletters I like</h2><br />
 <br />
 <span>This is a mix of tech and non-tech newsletters I am subscribed to. In random order:</span><br />
 <br />
 <ul>
-<li>byteSizeGo</li>
-<li>Andreas Brandhorst Newsletter (Sci-Fi author)</li>
+<li>Ruby Weekly</li>
+<li>The Pragmatic Engineer</li>
+<li>The Valuable Dev</li>
 <li>Changelog News</li>
-<li>Golang Weekly</li>
+<li>Andreas Brandhorst Newsletter (Sci-Fi author)</li>
 <li>Register Spill</li>
-<li>The Valuable Dev</li>
-<li>Monospace Mentor</li>
+<li>byteSizeGo</li>
+<li>The Imperfectionist</li>
+<li>Golang Weekly</li>
 <li>VK Newsletter</li>
+<li>Monospace Mentor</li>
 <li>Applied Go Weekly Newsletter</li>
-<li>Ruby Weekly</li>
-<li>The Pragmatic Engineer</li>
-<li>The Imperfectionist</li>
 </ul><br />
 <h2 style='display: inline' id='magazines-i-liked'>Magazines I like(d)</h2><br />
 <br />
 <span>This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order:</span><br />
 <br />
 <ul>
-<li>freeX (not published anymore)</li>
-<li>Linux Magazine</li>
 <li>Linux User</li>
+<li>Linux Magazine</li>
 <li>LWN (online only)</li>
+<li>freeX (not published anymore)</li>
 </ul><br />
 <h1 style='display: inline' id='formal-education'>Formal education</h1><br />
 <br />
diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
index 70a8e44f..e17fdf7d 100644
--- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
+++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
@@ -33,6 +33,7 @@
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-7-k3s-and-first-pod-deployments'>f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#updating'>Updating</a></li>
 <li>⇢ <a href='#installing-k3s'>Installing k3s</a></li>
 <li>⇢ ⇢ <a href='#generating-k3stoken-and-starting-the-first-k3s-node'>Generating <span class='inlinecode'>K3S_TOKEN</span> and starting the first k3s node</a></li>
@@ -63,6 +64,29 @@
 <br />
 <a class='textlink' href='https://k3s.io'>https://k3s.io</a><br />
 <br />
+<h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
+<br />
+<span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests and Helm charts in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
+<br />
+<span>**To view the exact manifests and charts as they existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
+$ cd f3s/
+</pre>
+<br />
+<span>**Current master branch** contains the ArgoCD-managed versions with:</span><br />
+<span>- Application manifests organized under <span class='inlinecode'>argocd-apps/{monitoring,services,infra,test}/</span></span><br />
+<span>- Additional resources under <span class='inlinecode'>*/manifests/</span> directories (e.g., <span class='inlinecode'>prometheus/manifests/</span>)</span><br />
+<span>- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands</span><br />
+<br />
+<span>The deployment concepts and architecture remain the same—only the deployment method changed from imperative (<span class='inlinecode'>helm install/upgrade</span>) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series.</span><br />
+<br />
 <h2 style='display: inline' id='updating'>Updating</h2><br />
 <br />
 <span>Before proceeding, I bring all systems involved up-to-date. On all three Rocky Linux 9 boxes <span class='inlinecode'>r0</span>, <span class='inlinecode'>r1</span>, and <span class='inlinecode'>r2</span>:</span><br />
@@ -878,21 +902,7 @@ http://www.gnu.org/software/src-highlite -->
 <br />
 <a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
 <br />
-<span>**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests and Helm charts in the repository have been reorganized for declarative deployment. To view the exact manifests and charts as they existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
-$ cd f3s/
-</pre>
-<br />
-<span>The current master branch contains the ArgoCD-managed versions with manifests organized under <span class='inlinecode'>argocd-apps/</span> and <span class='inlinecode'>*/manifests/</span> directories.</span><br />
-<br />
-<span>Within that repo, the <span class='inlinecode'>examples/conf/f3s/registry/</span> directory contains the Helm chart, a <span class='inlinecode'>Justfile</span>, and a detailed <span class='inlinecode'>README</span>. Here&#39;s the condensed walkthrough I used to roll out the registry with Helm.</span><br />
+<span>Within that repo, the <span class='inlinecode'>f3s/registry/</span> directory contains the Helm chart, a <span class='inlinecode'>Justfile</span>, and a detailed <span class='inlinecode'>README</span>. Here&#39;s the condensed walkthrough I used to roll out the registry with Helm.</span><br />
 <br />
 <h3 style='display: inline' id='prepare-the-nfs-backed-storage'>Prepare the NFS-backed storage</h3><br />
 <br />
diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
index 00fcadc9..0295f7dc 100644
--- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
+++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
@@ -33,6 +33,7 @@
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-8-observability'>f3s: Kubernetes with FreeBSD - Part 8: Observability</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#persistent-storage-recap'>Persistent storage recap</a></li>
 <li>⇢ <a href='#the-monitoring-namespace'>The monitoring namespace</a></li>
 <li>⇢ <a href='#installing-prometheus-and-grafana'>Installing Prometheus and Grafana</a></li>
@@ -74,7 +75,30 @@
 <br />
 <span>All manifests for the f3s stack live in my configuration repository:</span><br />
 <br />
-<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s </a><br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
+<br />
+<h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
+<br />
+<span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
+<br />
+<span>**To view the exact configuration as it existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
+$ cd f3s/prometheus/
+</pre>
+<br />
+<span>**Current master branch** contains the ArgoCD-managed versions with:</span><br />
+<span>- Application manifests organized under <span class='inlinecode'>argocd-apps/{monitoring,services,infra,test}/</span></span><br />
+<span>- Resources organized under <span class='inlinecode'>prometheus/manifests/</span>, <span class='inlinecode'>loki/</span>, etc.</span><br />
+<span>- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands</span><br />
+<br />
+<span>The deployment concepts and architecture remain the same—only the deployment method changed from imperative (<span class='inlinecode'>helm install/upgrade</span>) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series. </span><br />
 <br />
 <h2 style='display: inline' id='persistent-storage-recap'>Persistent storage recap</h2><br />
 <br />
@@ -107,20 +131,6 @@ namespace/monitoring created
 <br />
 <h2 style='display: inline' id='installing-prometheus-and-grafana'>Installing Prometheus and Grafana</h2><br />
 <br />
-<span>**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for declarative deployment. To view the exact configuration as it existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
-$ cd f3s/prometheus/
-</pre>
-<br />
-<span>The current master branch contains the ArgoCD-managed versions with Application manifests under <span class='inlinecode'>argocd-apps/</span> and resources organized under <span class='inlinecode'>prometheus/manifests/</span>, <span class='inlinecode'>loki/</span>, etc. The Justfiles have been updated to trigger ArgoCD syncs instead of direct Helm commands.</span><br />
-<br />
 <span>Prometheus and Grafana are deployed together using the <span class='inlinecode'>kube-prometheus-stack</span> Helm chart from the Prometheus community. This chart bundles Prometheus, Grafana, Alertmanager, and various exporters (Node Exporter, Kube State Metrics) into a single deployment. Ill explain what each component does in detail later when we look at the running pods.</span><br />
 <br />
 <h3 style='display: inline' id='prerequisites'>Prerequisites</h3><br />
diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
new file mode 100644
index 00000000..5d58d4c5
--- /dev/null
+++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
@@ -0,0 +1,1227 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD</title>
+<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
+<link rel="stylesheet" href="../style.css" />
+<link rel="stylesheet" href="style-override.css" />
+</head>
+<body>
+<p class="header">
+<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.gmi">Gemini</a>
+</p>
+<h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-x-gitops-with-argocd'>f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD</h1><br />
+<br />
+<span class='quote'>DRAFT - Not yet published</span><br />
+<br />
+<span>This is part X of the f3s series for my self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.</span><br />
+<br />
+<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
+<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
+<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
+<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
+<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
+<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
+<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br />
+<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br />
+<br />
+<a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br />
+<br />
+<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
+<br />
+<ul>
+<li><a href='#f3s-kubernetes-with-freebsd---part-x-gitops-with-argocd'>f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD</a></li>
+<li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#what-is-gitops'>What is GitOps?</a></li>
+<li>⇢ <a href='#what-is-argocd'>What is ArgoCD?</a></li>
+<li>⇢ <a href='#why-argocd-for-f3s'>Why ArgoCD for f3s?</a></li>
+<li>⇢ <a href='#deploying-argocd'>Deploying ArgoCD</a></li>
+<li>⇢ ⇢ <a href='#prerequisites'>Prerequisites</a></li>
+<li>⇢ ⇢ <a href='#installing-argocd'>Installing ArgoCD</a></li>
+<li>⇢ ⇢ <a href='#accessing-argocd'>Accessing ArgoCD</a></li>
+<li>⇢ <a href='#argocd-application-structure'>ArgoCD Application Structure</a></li>
+<li>⇢ <a href='#repository-organization'>Repository Organization</a></li>
+<li>⇢ <a href='#migration-strategy-incremental-one-app-at-a-time'>Migration Strategy: Incremental, One App at a Time</a></li>
+<li>⇢ ⇢ <a href='#migration-phases'>Migration Phases</a></li>
+<li>⇢ <a href='#example-migration-miniflux'>Example Migration: Miniflux</a></li>
+<li>⇢ ⇢ <a href='#before-imperative-helm-deployment'>Before: Imperative Helm deployment</a></li>
+<li>⇢ ⇢ <a href='#after-declarative-gitops-with-argocd'>After: Declarative GitOps with ArgoCD</a></li>
+<li>⇢ ⇢ <a href='#migration-procedure'>Migration procedure</a></li>
+<li>⇢ <a href='#complex-migration-prometheus-with-multi-source'>Complex Migration: Prometheus with Multi-Source</a></li>
+<li>⇢ ⇢ <a href='#sync-waves-and-hooks'>Sync Waves and Hooks</a></li>
+<li>⇢ <a href='#migration-results'>Migration Results</a></li>
+<li>⇢ <a href='#benefits-realized'>Benefits Realized</a></li>
+<li>⇢ ⇢ <a href='#1-single-source-of-truth'>1. Single Source of Truth</a></li>
+<li>⇢ ⇢ <a href='#2-automatic-synchronization'>2. Automatic Synchronization</a></li>
+<li>⇢ ⇢ <a href='#3-drift-detection-and-self-healing'>3. Drift Detection and Self-Healing</a></li>
+<li>⇢ ⇢ <a href='#4-easy-rollbacks'>4. Easy Rollbacks</a></li>
+<li>⇢ ⇢ <a href='#5-disaster-recovery'>5. Disaster Recovery</a></li>
+<li>⇢ ⇢ <a href='#6-documentation-by-default'>6. Documentation by Default</a></li>
+<li>⇢ ⇢ <a href='#7-safe-experimentation'>7. Safe Experimentation</a></li>
+<li>⇢ <a href='#challenges-and-solutions'>Challenges and Solutions</a></li>
+<li>⇢ ⇢ <a href='#challenge-1-helm-release-adoption'>Challenge 1: Helm Release Adoption</a></li>
+<li>⇢ ⇢ <a href='#challenge-2-persistent-volumes-not-tracked-by-helm'>Challenge 2: Persistent Volumes Not Tracked by Helm</a></li>
+<li>⇢ ⇢ <a href='#challenge-3-secrets-management'>Challenge 3: Secrets Management</a></li>
+<li>⇢ ⇢ <a href='#challenge-4-grafana-not-reloading-datasources'>Challenge 4: Grafana Not Reloading Datasources</a></li>
+<li>⇢ ⇢ <a href='#challenge-5-prometheus-with-multiple-sources'>Challenge 5: Prometheus With Multiple Sources</a></li>
+<li>⇢ ⇢ <a href='#challenge-6-sync-ordering-for-prometheus'>Challenge 6: Sync Ordering for Prometheus</a></li>
+<li>⇢ <a href='#justfile-evolution'>Justfile Evolution</a></li>
+<li>⇢ <a href='#lessons-learned'>Lessons Learned</a></li>
+<li>⇢ <a href='#future-improvements'>Future Improvements</a></li>
+<li>⇢ ⇢ <a href='#1-external-secrets-operator'>1. External Secrets Operator</a></li>
+<li>⇢ ⇢ <a href='#2-applicationset-for-similar-apps'>2. ApplicationSet for Similar Apps</a></li>
+<li>⇢ ⇢ <a href='#3-app-of-apps-pattern'>3. App-of-Apps Pattern</a></li>
+<li>⇢ ⇢ <a href='#4-argocd-image-updater'>4. ArgoCD Image Updater</a></li>
+<li>⇢ <a href='#summary'>Summary</a></li>
+</ul><br />
+<h2 style='display: inline' id='introduction'>Introduction</h2><br />
+<br />
+<span>In the previous posts, I deployed applications to the k3s cluster using Helm charts and Justfiles—running <span class='inlinecode'>just install</span> or <span class='inlinecode'>just upgrade</span> to imperatively push changes to the cluster. While this approach works, it has several drawbacks:</span><br />
+<br />
+<ul>
+<li>**No single source of truth**: The cluster state depends on which commands were run and when</li>
+<li>**Manual synchronization**: Every change requires manually running commands</li>
+<li>**Drift detection is hard**: No easy way to know if cluster state matches the desired configuration</li>
+<li>**Rollback complexity**: Rolling back changes means re-running old Helm commands</li>
+<li>**No audit trail**: Hard to track who changed what and when</li>
+</ul><br />
+<span>This blog post covers the migration from imperative Helm deployments to declarative GitOps using ArgoCD. After this migration, the Git repository becomes the single source of truth, and ArgoCD automatically ensures the cluster matches what&#39;s defined in Git.</span><br />
+<br />
+<h2 style='display: inline' id='what-is-gitops'>What is GitOps?</h2><br />
+<br />
+<span>GitOps is an operational framework that applies DevOps best practices—like version control, collaboration, and CI/CD—to infrastructure automation. The core idea is simple: the entire desired state of your infrastructure is stored in Git, and automated processes ensure the actual state matches the desired state.</span><br />
+<br />
+<span>Key principles:</span><br />
+<br />
+<ul>
+<li>**Declarative**: The system&#39;s desired state is described declaratively (YAML manifests, Helm values)</li>
+<li>**Versioned and immutable**: All changes are committed to Git, providing a complete history</li>
+<li>**Pulled automatically**: An agent in the cluster continuously pulls the desired state from Git</li>
+<li>**Continuously reconciled**: The agent ensures the actual state matches the desired state, automatically correcting drift</li>
+</ul><br />
+<span>For Kubernetes, this means:</span><br />
+<br />
+<span>1. All manifests, Helm charts, and configuration live in a Git repository</span><br />
+<span>2. A tool (ArgoCD in our case) watches the repository</span><br />
+<span>3. When changes are pushed to Git, ArgoCD automatically applies them to the cluster</span><br />
+<span>4. If someone manually changes resources in the cluster, ArgoCD detects the drift and can automatically revert it</span><br />
+<br />
+<h2 style='display: inline' id='what-is-argocd'>What is ArgoCD?</h2><br />
+<br />
+<span>ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It&#39;s implemented as a Kubernetes controller that continuously monitors running applications and compares the current, live state against the desired target state defined in Git.</span><br />
+<br />
+<a class='textlink' href='https://argo-cd.readthedocs.io'>ArgoCD Documentation</a><br />
+<br />
+<span>Key features:</span><br />
+<br />
+<ul>
+<li>**Automated deployment**: Monitors Git repositories and automatically syncs changes to the cluster</li>
+<li>**Application definitions**: Defines applications as CRDs (Custom Resource Definitions)</li>
+<li>**Health assessment**: Understands Kubernetes resources and can determine if an application is healthy</li>
+<li>**Web UI and CLI**: Provides both a web interface and command-line tool for managing applications</li>
+<li>**RBAC**: Role-based access control for team collaboration</li>
+<li>**SSO integration**: Can integrate with existing authentication systems</li>
+<li>**Multi-cluster support**: Can manage applications across multiple Kubernetes clusters</li>
+<li>**Sync waves and hooks**: Control the order of resource deployment and run jobs at specific lifecycle points</li>
+</ul><br />
+<h2 style='display: inline' id='why-argocd-for-f3s'>Why ArgoCD for f3s?</h2><br />
+<br />
+<span>For a home lab cluster, ArgoCD provides several benefits:</span><br />
+<br />
+<span>**Disaster recovery**: If the entire cluster is lost, I can rebuild it by:</span><br />
+<span>1. Bootstrapping a new k3s cluster</span><br />
+<span>2. Installing ArgoCD</span><br />
+<span>3. Pointing ArgoCD at the Git repository</span><br />
+<span>4. All applications automatically deploy to the desired state</span><br />
+<br />
+<span>**Experimentation safety**: I can test changes in a separate Git branch without affecting the running cluster. Once validated, merge to master and ArgoCD applies the changes.</span><br />
+<br />
+<span>**Drift detection**: If I manually change something in the cluster (for debugging), ArgoCD shows the difference and can automatically revert it.</span><br />
+<br />
+<span>**Declarative configuration**: The Git repository documents the entire cluster configuration. No need to remember which <span class='inlinecode'>just</span> commands to run or in which order.</span><br />
+<br />
+<span>**Automatic sync**: Push to Git, and changes deploy automatically. No need to SSH to a workstation and run Helm commands.</span><br />
+<br />
+<h2 style='display: inline' id='deploying-argocd'>Deploying ArgoCD</h2><br />
+<br />
+<span>ArgoCD itself runs as a set of Kubernetes resources in the cluster. The official installation method uses <span class='inlinecode'>kubectl apply</span>, which is fitting—ArgoCD manages everything else via GitOps, but ArgoCD itself needs a bootstrap.</span><br />
+<br />
+<h3 style='display: inline' id='prerequisites'>Prerequisites</h3><br />
+<br />
+<span>Create the <span class='inlinecode'>cicd</span> namespace where ArgoCD will run:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl create namespace cicd
+namespace/cicd created
+</pre>
+<br />
+<h3 style='display: inline' id='installing-argocd'>Installing ArgoCD</h3><br />
+<br />
+<span>The ArgoCD installation lives in the configuration repository:</span><br />
+<br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/argocd'>codeberg.org/snonux/conf/f3s/argocd</a><br />
+<br />
+<span>I deployed ArgoCD using Helm instead of the raw manifests. This provides easier upgrades and customization. The installation is managed via a Justfile:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ cd conf/f3s/argocd
+$ just install
+helm repo add argo https://argoproj.github.io/argo-helm
+helm repo update
+helm install argocd argo/argo-cd \
+    --namespace cicd \
+    --version <font color="#000000">7.7</font>.<font color="#000000">12</font> \
+    -f values.yaml
+NAME: argocd
+LAST DEPLOYED: ...
+NAMESPACE: cicd
+STATUS: deployed
+</pre>
+<br />
+<span>The <span class='inlinecode'>values.yaml</span> file configures several important aspects:</span><br />
+<br />
+<span>**Persistent storage for the repo-server**: ArgoCD clones Git repositories to cache them locally. I configured a persistent volume so the cache survives pod restarts:</span><br />
+<br />
+<pre>
+repoServer:
+  volumes:
+    - name: repo-cache
+      persistentVolumeClaim:
+        claimName: argocd-repo-cache-pvc
+  volumeMounts:
+    - name: repo-cache
+      mountPath: /tmp
+</pre>
+<br />
+<span>**Admin password preservation**: By default, the admin password is auto-generated and stored in a secret. To ensure it persists across Helm upgrades:</span><br />
+<br />
+<pre>
+configs:
+  secret:
+    createSecret: false
+</pre>
+<br />
+<span>I manually created the secret before installation:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ ARGOCD_ADMIN_PASSWORD=$(pwgen -s <font color="#000000">32</font> <font color="#000000">1</font>)
+$ BCRYPT_HASH=$(htpasswd -nbBC <font color="#000000">10</font> <font color="#808080">""</font> <font color="#808080">"$ARGOCD_ADMIN_PASSWORD"</font> | tr -d <font color="#808080">':</font>\n<font color="#808080">'</font> | sed <font color="#808080">'s/$2y/$2a/'</font>)
+$ kubectl create secret generic argocd-secret \
+    --from-literal=admin.password=<font color="#808080">"$BCRYPT_HASH"</font> \
+    -n cicd
+$ echo <font color="#808080">"ArgoCD admin password: $ARGOCD_ADMIN_PASSWORD"</font>
+</pre>
+<br />
+<span>**Server configuration**: Enabled insecure mode since TLS is handled by the OpenBSD edge relays:</span><br />
+<br />
+<pre>
+server:
+  insecure: true
+</pre>
+<br />
+<h3 style='display: inline' id='accessing-argocd'>Accessing ArgoCD</h3><br />
+<br />
+<span>After deployment, ArgoCD runs several pods in the <span class='inlinecode'>cicd</span> namespace:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get pods -n cicd
+NAME                                                READY   STATUS    RESTARTS   AGE
+argocd-application-controller-<font color="#000000">0</font>                     <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-applicationset-controller-66d6b9b8f4-vhm9k   <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-dex-server-7fb556b7dd-xjr2l                  <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-notifications-controller-6d8dd4c5f5-b8vwl    <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-redis-77b8d6c6d4-mz9hg                       <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-repo-server-5f98f77b97-8xtcq                 <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-server-6b9c4b4f8d-kxw7p                      <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+</pre>
+<br />
+<span>I created an ingress to expose the ArgoCD web UI:</span><br />
+<br />
+<pre>
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: argocd-server-ingress
+  namespace: cicd
+  annotations:
+    spec.ingressClassName: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: web
+spec:
+  rules:
+    - host: argocd.f3s.buetow.org
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: argocd-server
+                port:
+                  number: 80
+</pre>
+<br />
+<span>Following the same pattern as other services, the OpenBSD edge relays terminate TLS and forward traffic through WireGuard to the cluster. ArgoCD is now accessible at:</span><br />
+<br />
+<a class='textlink' href='https://argocd.f3s.buetow.org'>ArgoCD Web UI</a><br />
+<br />
+<span>The ArgoCD CLI can also be used for operations:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ argocd login argocd.f3s.buetow.org
+$ argocd app list
+</pre>
+<br />
+<h2 style='display: inline' id='argocd-application-structure'>ArgoCD Application Structure</h2><br />
+<br />
+<span>ArgoCD uses a CRD called <span class='inlinecode'>Application</span> to define what should be deployed. Each application specifies:</span><br />
+<br />
+<ul>
+<li>**Source**: Where the manifests live (Git repo, Helm chart repository, or both)</li>
+<li>**Destination**: Which cluster and namespace to deploy to</li>
+<li>**Sync policy**: Whether to automatically sync changes</li>
+</ul><br />
+<span>Here&#39;s a simple example for the miniflux application:</span><br />
+<br />
+<pre>
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: miniflux
+  namespace: cicd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    repoURL: https://codeberg.org/snonux/conf.git
+    targetRevision: master
+    path: f3s/miniflux/helm-chart
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: services
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=false
+    retry:
+      limit: 3
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 1m
+</pre>
+<br />
+<span>Key fields:</span><br />
+<br />
+<ul>
+<li><span class='inlinecode'>source.path</span>: Points to the Helm chart directory in Git</li>
+<li><span class='inlinecode'>destination.namespace</span>: Where to deploy the application</li>
+<li><span class='inlinecode'>syncPolicy.automated.prune</span>: Delete resources that are removed from Git</li>
+<li><span class='inlinecode'>syncPolicy.automated.selfHeal</span>: Automatically revert manual changes in the cluster</li>
+<li><span class='inlinecode'>finalizers</span>: Ensures ArgoCD deletes all resources when the Application is deleted</li>
+</ul><br />
+<h2 style='display: inline' id='repository-organization'>Repository Organization</h2><br />
+<br />
+<span>I reorganized the configuration repository to support GitOps:</span><br />
+<br />
+<pre>
+/home/paul/git/conf/f3s/
+├── argocd-apps/               # ArgoCD Application manifests (organized by namespace)
+│   ├── README.md              # Documentation of structure
+│   ├── monitoring/            # Observability stack (6 apps)
+│   │   ├── alloy.yaml
+│   │   ├── grafana-ingress.yaml
+│   │   ├── loki.yaml
+│   │   ├── prometheus.yaml
+│   │   ├── pushgateway.yaml
+│   │   └── tempo.yaml
+│   ├── services/              # User-facing applications (13 apps)
+│   │   ├── anki-sync-server.yaml
+│   │   ├── audiobookshelf.yaml
+│   │   ├── filebrowser.yaml
+│   │   ├── immich.yaml
+│   │   ├── keybr.yaml
+│   │   ├── kobo-sync-server.yaml
+│   │   ├── miniflux.yaml
+│   │   ├── opodsync.yaml
+│   │   ├── radicale.yaml
+│   │   ├── syncthing.yaml
+│   │   ├── tracing-demo.yaml
+│   │   ├── wallabag.yaml
+│   │   └── webdav.yaml
+│   ├── infra/                 # Infrastructure services (1 app)
+│   │   └── registry.yaml
+│   └── test/                  # Test/example applications (1 app)
+│       └── example-apache-volume-claim.yaml
+├── miniflux/                  # Application directories (unchanged)
+│   ├── helm-chart/
+│   │   ├── Chart.yaml
+│   │   ├── values.yaml
+│   │   └── templates/
+│   └── Justfile               # Updated for ArgoCD
+├── prometheus/
+│   ├── manifests/             # NEW: Additional manifests
+│   │   ├── persistent-volumes.yaml
+│   │   ├── grafana-restart-hook.yaml
+│   │   ├── freebsd-recording-rules.yaml
+│   │   └── ...
+│   └── Justfile               # Updated for ArgoCD
+└── ...
+</pre>
+<br />
+<span>The application directories (miniflux, prometheus, etc.) remained mostly unchanged—ArgoCD references the same Helm charts. The main additions:</span><br />
+<br />
+<span>1. **argocd-apps/**: Application manifests organized by Kubernetes namespace for better clarity</span><br />
+<span>   - <span class='inlinecode'>monitoring/</span>: 6 observability applications</span><br />
+<span>   - <span class='inlinecode'>services/</span>: 13 user-facing applications</span><br />
+<span>   - <span class='inlinecode'>infra/</span>: 1 infrastructure application (registry)</span><br />
+<span>   - <span class='inlinecode'>test/</span>: 1 test application</span><br />
+<span>2. ***/manifests/**: Additional Kubernetes manifests for complex apps (like Prometheus)</span><br />
+<span>3. **Justfiles updated**: Changed from <span class='inlinecode'>helm install/upgrade</span> to <span class='inlinecode'>argocd app sync</span></span><br />
+<br />
+<span>This organization makes it easy to apply all applications in a specific namespace or manage them independently.</span><br />
+<br />
+<h2 style='display: inline' id='migration-strategy-incremental-one-app-at-a-time'>Migration Strategy: Incremental, One App at a Time</h2><br />
+<br />
+<span>Rather than attempting a "big bang" migration of all 21 applications at once, I migrated them incrementally:</span><br />
+<br />
+<span>1. **Start with a simple app**: Validate the pattern with a low-risk application</span><br />
+<span>2. **Migrate in waves**: Group similar applications and migrate together</span><br />
+<span>3. **Validate thoroughly**: Ensure each app is healthy before moving to the next</span><br />
+<span>4. **Learn and iterate**: Apply lessons from earlier migrations to later ones</span><br />
+<br />
+<span>This approach reduced risk and allowed me to refine the migration process.</span><br />
+<br />
+<h3 style='display: inline' id='migration-phases'>Migration Phases</h3><br />
+<br />
+<span>**Phase 1: Simple services** (13 apps)</span><br />
+<ul>
+<li>miniflux, freshrss, wallabag</li>
+<li>anki-sync-server, kobo-sync-server, opodsync</li>
+<li>radicale, syncthing, audiobookshelf</li>
+<li>filebrowser, keybr, webdav</li>
+<li>example-apache, example-apache-volume-claim</li>
+</ul><br />
+<span>These apps have straightforward Helm charts with no complex dependencies. Pattern established:</span><br />
+<span>1. Create Application manifest in <span class='inlinecode'>argocd-apps/</span></span><br />
+<span>2. Apply with <span class='inlinecode'>kubectl apply -f argocd-apps/&lt;app&gt;.yaml</span></span><br />
+<span>3. Verify sync status: <span class='inlinecode'>argocd app get &lt;app&gt;</span></span><br />
+<span>4. Update Justfile to use ArgoCD commands</span><br />
+<br />
+<span>**Phase 2: Infrastructure apps** (3 apps)</span><br />
+<ul>
+<li>registry (Docker image registry)</li>
+<li>pushgateway (Prometheus metrics ingestion)</li>
+<li>immich (photo management with complex dependencies)</li>
+</ul><br />
+<span>**Phase 3: Monitoring stack** (4 apps)</span><br />
+<ul>
+<li>tempo (distributed tracing)</li>
+<li>loki (log aggregation)</li>
+<li>alloy (log collection)</li>
+<li>prometheus (metrics and monitoring)</li>
+</ul><br />
+<span>**Phase 4: Monitoring addons** (1 app)</span><br />
+<ul>
+<li>grafana-ingress (separate ingress for Grafana)</li>
+</ul><br />
+<h2 style='display: inline' id='example-migration-miniflux'>Example Migration: Miniflux</h2><br />
+<br />
+<span>Let me walk through the migration of miniflux as a concrete example.</span><br />
+<br />
+<h3 style='display: inline' id='before-imperative-helm-deployment'>Before: Imperative Helm deployment</h3><br />
+<br />
+<span>Original Justfile:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>NAMESPACE := <font color="#808080">"services"</font>
+APP_NAME := <font color="#808080">"miniflux"</font>
+
+install:
+    kubectl apply -f helm-chart/persistent-volumes.yaml
+    helm install {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}
+
+upgrade:
+    helm upgrade {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}
+
+uninstall:
+    helm uninstall {{APP_NAME}} --namespace {{NAMESPACE}}
+    kubectl delete -f helm-chart/persistent-volumes.yaml
+
+status:
+    @kubectl get all -n {{NAMESPACE}} -l app={{APP_NAME}}
+</pre>
+<br />
+<span>Workflow:</span><br />
+<span>1. Make changes to <span class='inlinecode'>helm-chart/</span></span><br />
+<span>2. Run <span class='inlinecode'>just upgrade</span></span><br />
+<span>3. Helm pushes changes to cluster</span><br />
+<br />
+<h3 style='display: inline' id='after-declarative-gitops-with-argocd'>After: Declarative GitOps with ArgoCD</h3><br />
+<br />
+<span>Created <span class='inlinecode'>argocd-apps/services/miniflux.yaml</span>:</span><br />
+<br />
+<pre>
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: miniflux
+  namespace: cicd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    repoURL: https://codeberg.org/snonux/conf.git
+    targetRevision: master
+    path: f3s/miniflux/helm-chart
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: services
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=false
+    retry:
+      limit: 3
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 1m
+</pre>
+<br />
+<span>Updated Justfile:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>NAMESPACE := <font color="#808080">"services"</font>
+APP_NAME := <font color="#808080">"miniflux"</font>
+
+status:
+    @echo <font color="#808080">"=== Pods ==="</font>
+    @kubectl get pods -n {{NAMESPACE}} -l app={{APP_NAME}}
+    @echo <font color="#808080">""</font>
+    @echo <font color="#808080">"=== Services ==="</font>
+    @kubectl get svc -n {{NAMESPACE}} -l app={{APP_NAME}}
+    @echo <font color="#808080">""</font>
+    @echo <font color="#808080">"=== ArgoCD Status ==="</font>
+    @kubectl get application {{APP_NAME}} -n cicd -o jsonpath=<font color="#808080">'Sync: {.status.sync.status}, Health: {.status.health.status}'</font> <font color="#000000">2</font>&gt;/dev/null &amp;&amp; echo <font color="#808080">""</font>
+
+sync:
+    @echo <font color="#808080">"Triggering ArgoCD sync..."</font>
+    @kubectl annotate application {{APP_NAME}} -n cicd argocd.argoproj.io/refresh=normal --overwrite
+    @sleep <font color="#000000">2</font>
+    @kubectl get application {{APP_NAME}} -n cicd -o jsonpath=<font color="#808080">'Sync: {.status.sync.status}, Health: {.status.health.status}'</font> &amp;&amp; echo <font color="#808080">""</font>
+
+argocd-status:
+    argocd app get {{APP_NAME}} --core
+
+logs:
+    kubectl logs -n {{NAMESPACE}} -l app={{APP_NAME}} --tail=<font color="#000000">100</font> -f
+</pre>
+<br />
+<span>New workflow:</span><br />
+<span>1. Make changes to <span class='inlinecode'>helm-chart/</span></span><br />
+<span>2. Commit and push to Git</span><br />
+<span>3. ArgoCD automatically detects and syncs changes</span><br />
+<span>4. (Optional) Run <span class='inlinecode'>just sync</span> to force immediate sync</span><br />
+<br />
+<h3 style='display: inline' id='migration-procedure'>Migration procedure</h3><br />
+<br />
+<span>1. **Backup current state**:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ helm get values miniflux -n services &gt; /tmp/miniflux-backup-values.yaml
+$ kubectl get all,ingress -n services -o yaml &gt; /tmp/miniflux-backup.yaml
+</pre>
+<br />
+<span>2. **Create Application manifest**:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl apply -f argocd-apps/services/miniflux.yaml
+application.argoproj.io/miniflux created
+</pre>
+<br />
+<span>3. **Verify ArgoCD adopted the resources**:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ argocd app get miniflux
+Name:               miniflux
+Project:            default
+Server:             https://kubernetes.default.svc
+Namespace:          services
+URL:                https://argocd.f3s.buetow.org/applications/miniflux
+Repo:               https://codeberg.org/snonux/conf.git
+Target:             master
+Path:               f3s/miniflux/helm-chart
+SyncWindow:         Sync Allowed
+Sync Policy:        Automated (Prune)
+Sync Status:        Synced to master (4e3c216)
+Health Status:      Healthy
+</pre>
+<br />
+<span>4. **Monitor for issues**:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get pods -n services -l app=miniflux -w
+NAME                                READY   STATUS    RESTARTS   AGE
+miniflux-postgres-556444cb8d-xvv2p  <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          54d
+miniflux-server-85d7c64664-stmt<font color="#000000">9</font>    <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          54d
+</pre>
+<br />
+<span>5. **Test the application**:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ curl -I https://flux.f3s.buetow.org
+HTTP/<font color="#000000">2</font> <font color="#000000">200</font>
+</pre>
+<br />
+<span>6. **Update Justfile** and commit changes</span><br />
+<br />
+<span>Total time: 10 minutes. Zero downtime.</span><br />
+<br />
+<h2 style='display: inline' id='complex-migration-prometheus-with-multi-source'>Complex Migration: Prometheus with Multi-Source</h2><br />
+<br />
+<span>The Prometheus migration was more complex because it combines:</span><br />
+<ul>
+<li>Upstream Helm chart (kube-prometheus-stack)</li>
+<li>Custom manifests (PersistentVolumes, recording rules, dashboards)</li>
+<li>Sync hooks (PostSync job to restart Grafana)</li>
+</ul><br />
+<span>ArgoCD supports "multi-source" Applications that combine multiple sources:</span><br />
+<br />
+<pre>
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: prometheus
+  namespace: cicd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  sources:
+    # Source 1: Upstream Helm chart from prometheus-community
+    - repoURL: https://prometheus-community.github.io/helm-charts
+      chart: kube-prometheus-stack
+      targetRevision: 55.5.0
+      helm:
+        releaseName: prometheus
+        valuesObject:
+          # Full Prometheus configuration embedded here
+          kubeEtcd:
+            enabled: true
+            endpoints:
+              - 192.168.2.120
+              - 192.168.2.121
+              - 192.168.2.122
+          # ... (hundreds of lines of configuration)
+
+    # Source 2: Additional manifests from Git repository
+    - repoURL: https://codeberg.org/snonux/conf.git
+      targetRevision: master
+      path: f3s/prometheus/manifests
+
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: monitoring
+
+  syncPolicy:
+    automated:
+      prune: false  # Manual pruning for safety on complex stack
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=false
+      - ServerSideApply=true
+    retry:
+      limit: 3
+      backoff:
+        duration: 10s
+        factor: 2
+        maxDuration: 3m
+</pre>
+<br />
+<span>The <span class='inlinecode'>prometheus/manifests/</span> directory contains:</span><br />
+<br />
+<pre>
+f3s/prometheus/manifests/
+├── persistent-volumes.yaml              # Sync wave 0
+├── additional-scrape-configs-secret.yaml # Sync wave 1
+├── grafana-datasources-configmap.yaml   # Sync wave 1
+├── freebsd-recording-rules.yaml         # Sync wave 3
+├── openbsd-recording-rules.yaml         # Sync wave 3
+├── zfs-recording-rules.yaml             # Sync wave 3
+├── epimetheus-dashboard.yaml            # Sync wave 4
+├── zfs-dashboards.yaml                  # Sync wave 4
+├── grafana-restart-hook.yaml            # Sync wave 10 (PostSync)
+└── grafana-restart-rbac.yaml            # Sync wave 0
+</pre>
+<br />
+<h3 style='display: inline' id='sync-waves-and-hooks'>Sync Waves and Hooks</h3><br />
+<br />
+<span>ArgoCD allows controlling the order of resource deployment using sync waves (the <span class='inlinecode'>argocd.argoproj.io/sync-wave</span> annotation):</span><br />
+<br />
+<ul>
+<li>**Wave 0**: Infrastructure (PersistentVolumes, RBAC)</li>
+<li>**Wave 1**: Configuration (Secrets, ConfigMaps)</li>
+<li>**Wave 3**: Recording rules (PrometheusRule CRDs)</li>
+<li>**Wave 4**: Dashboards (ConfigMaps with <span class='inlinecode'>grafana_dashboard: &#39;1&#39;</span> label)</li>
+<li>**Wave 10**: PostSync hooks (Jobs that run after everything else)</li>
+</ul><br />
+<span>The Grafana restart hook ensures Grafana reloads datasources after they&#39;re updated:</span><br />
+<br />
+<pre>
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: grafana-restart-hook
+  namespace: monitoring
+  annotations:
+    argocd.argoproj.io/hook: PostSync
+    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
+    argocd.argoproj.io/sync-wave: "10"
+spec:
+  template:
+    spec:
+      serviceAccountName: grafana-restart-sa
+      restartPolicy: OnFailure
+      containers:
+        - name: kubectl
+          image: bitnami/kubectl:latest
+          command:
+            - /bin/sh
+            - -c
+            - |
+              kubectl wait --for=condition=available --timeout=300s deployment/prometheus-grafana -n monitoring || true
+              kubectl delete pod -n monitoring -l app.kubernetes.io/name=grafana --ignore-not-found=true
+  backoffLimit: 2
+</pre>
+<br />
+<span>This replaces the manual step in the old Justfile that required running <span class='inlinecode'>kubectl delete pod</span> after every upgrade.</span><br />
+<br />
+<h2 style='display: inline' id='migration-results'>Migration Results</h2><br />
+<br />
+<span>After migrating all 21 applications to ArgoCD:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ argocd app list
+NAME                      CLUSTER                         NAMESPACE    PROJECT  STATUS  HEALTH   SYNCPOLICY
+alloy                     https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
+anki-sync-server          https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+audiobookshelf            https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+example-apache            https://kubernetes.default.svc  <b><u><font color="#000000">test</font></u></b>         default  Synced  Healthy  Auto-Prune
+example-apache-volume-... https://kubernetes.default.svc  <b><u><font color="#000000">test</font></u></b>         default  Synced  Healthy  Auto-Prune
+filebrowser               https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+freshrss                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+grafana-ingress           https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
+immich                    https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+keybr                     https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+kobo-sync-server          https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+loki                      https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
+miniflux                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+opodsync                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+prometheus                https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto
+pushgateway               https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
+radicale                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+registry                  https://kubernetes.default.svc  infra        default  Synced  Healthy  Auto-Prune
+syncthing                 https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+tempo                     https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
+wallabag                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+webdav                    https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+</pre>
+<br />
+<span>All 21 applications: **Synced** and **Healthy**.</span><br />
+<br />
+<span>ArgoCD Web UI:</span><br />
+<br />
+<a href='./f3s-kubernetes-with-freebsd-part-X/argocd-apps-list.png'><img alt='ArgoCD Applications List' title='ArgoCD Applications List' src='./f3s-kubernetes-with-freebsd-part-X/argocd-apps-list.png' /></a><br />
+<br />
+<a href='./f3s-kubernetes-with-freebsd-part-X/argocd-app-tree.png'><img alt='ArgoCD Application Resource Tree' title='ArgoCD Application Resource Tree' src='./f3s-kubernetes-with-freebsd-part-X/argocd-app-tree.png' /></a><br />
+<br />
+<h2 style='display: inline' id='benefits-realized'>Benefits Realized</h2><br />
+<br />
+<h3 style='display: inline' id='1-single-source-of-truth'>1. Single Source of Truth</h3><br />
+<br />
+<span>The Git repository at <span class='inlinecode'>https://codeberg.org/snonux/conf</span> now contains the complete cluster configuration. Anyone can clone it and see exactly what&#39;s deployed:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf/f3s
+$ ls argocd-apps/
+alloy.yaml  anki-sync-server.yaml  audiobookshelf.yaml  ...
+</pre>
+<br />
+<h3 style='display: inline' id='2-automatic-synchronization'>2. Automatic Synchronization</h3><br />
+<br />
+<span>Push to Git, and changes deploy automatically:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ cd conf/f3s/miniflux/helm-chart
+$ vim values.yaml  <i><font color="silver"># Change replica count from 1 to 2</font></i>
+$ git add values.yaml
+$ git commit -m <font color="#808080">"Scale miniflux to 2 replicas"</font>
+$ git push
+<i><font color="silver"># ArgoCD detects change within 3 minutes and syncs automatically</font></i>
+</pre>
+<br />
+<span>No need to SSH to a workstation, pull the repo, and run <span class='inlinecode'>just upgrade</span>.</span><br />
+<br />
+<h3 style='display: inline' id='3-drift-detection-and-self-healing'>3. Drift Detection and Self-Healing</h3><br />
+<br />
+<span>If someone manually changes a resource in the cluster, ArgoCD detects it:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl scale deployment miniflux-server -n services --replicas=<font color="#000000">3</font>
+deployment.apps/miniflux-server scaled
+
+<i><font color="silver"># ArgoCD detects drift within 3 minutes</font></i>
+$ argocd app get miniflux
+...
+Sync Status:        OutOfSync from master (4e3c216)
+</pre>
+<br />
+<span>With <span class='inlinecode'>selfHeal: true</span>, ArgoCD automatically reverts the change back to 2 replicas (the value in Git).</span><br />
+<br />
+<h3 style='display: inline' id='4-easy-rollbacks'>4. Easy Rollbacks</h3><br />
+<br />
+<span>To rollback a change:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git revert HEAD
+$ git push
+<i><font color="silver"># ArgoCD automatically rolls back to the previous state</font></i>
+</pre>
+<br />
+<span>Or rollback to a specific commit:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ argocd app rollback miniflux &lt;revision-id&gt;
+</pre>
+<br />
+<h3 style='display: inline' id='5-disaster-recovery'>5. Disaster Recovery</h3><br />
+<br />
+<span>If the entire cluster is destroyed, recovery is straightforward:</span><br />
+<br />
+<span>1. Bootstrap a new k3s cluster</span><br />
+<span>2. Create namespaces</span><br />
+<span>3. Install ArgoCD</span><br />
+<span>4. Apply all Application manifests:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl apply -f argocd-apps/
+</pre>
+<span>5. ArgoCD deploys all 21 applications to their desired state</span><br />
+<br />
+<span>Total recovery time: ~30 minutes (mostly waiting for pods to pull images and start).</span><br />
+<br />
+<h3 style='display: inline' id='6-documentation-by-default'>6. Documentation by Default</h3><br />
+<br />
+<span>The Application manifests serve as documentation:</span><br />
+<br />
+<ul>
+<li>Which Helm chart version is deployed? → Check <span class='inlinecode'>targetRevision</span></li>
+<li>What custom values are configured? → Check <span class='inlinecode'>valuesObject</span></li>
+<li>Which namespace does this deploy to? → Check <span class='inlinecode'>destination.namespace</span></li>
+<li>Is auto-sync enabled? → Check <span class='inlinecode'>syncPolicy.automated</span></li>
+</ul><br />
+<span>No more guessing or checking <span class='inlinecode'>helm list</span> output.</span><br />
+<br />
+<h3 style='display: inline' id='7-safe-experimentation'>7. Safe Experimentation</h3><br />
+<br />
+<span>Create a feature branch, make changes, and preview them:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git checkout -b test-prometheus-upgrade
+$ vim argocd-apps/prometheus.yaml  <i><font color="silver"># Bump chart version</font></i>
+$ git commit -am <font color="#808080">"Test Prometheus 56.0.0"</font>
+$ git push origin test-prometheus-upgrade
+
+<i><font color="silver"># Temporarily point ArgoCD at the feature branch</font></i>
+$ kubectl patch application prometheus -n cicd \
+    --type merge \
+    -p <font color="#808080">'{"spec":{"source":{"targetRevision":"test-prometheus-upgrade"}}}'</font>
+
+<i><font color="silver"># Verify changes in ArgoCD Web UI</font></i>
+<i><font color="silver"># If good: merge to master</font></i>
+<i><font color="silver"># If bad: revert the patch</font></i>
+</pre>
+<br />
+<h2 style='display: inline' id='challenges-and-solutions'>Challenges and Solutions</h2><br />
+<br />
+<h3 style='display: inline' id='challenge-1-helm-release-adoption'>Challenge 1: Helm Release Adoption</h3><br />
+<br />
+<span>When creating an Application for an existing Helm release, ArgoCD needs to "adopt" the resources. This failed initially with errors like:</span><br />
+<br />
+<pre>
+The Helm operation failed with an error: release miniflux failed, and has been uninstalled due to atomic being set: timed out waiting for the condition
+</pre>
+<br />
+<span>**Solution**: For existing Helm releases, I first ensured the Application manifest matched the current Helm values exactly. ArgoCD then recognized the resources were already in the desired state and adopted them without re-deploying.</span><br />
+<br />
+<h3 style='display: inline' id='challenge-2-persistent-volumes-not-tracked-by-helm'>Challenge 2: Persistent Volumes Not Tracked by Helm</h3><br />
+<br />
+<span>PersistentVolumes are cluster-scoped resources, not namespace-scoped. Many of my Helm charts created PVs using <span class='inlinecode'>kubectl apply -f persistent-volumes.yaml</span> outside of Helm.</span><br />
+<br />
+<span>**Solution**: For simple apps, I moved the PV definitions into the Helm chart templates. For complex apps (like Prometheus), I used the multi-source pattern with PVs in the <span class='inlinecode'>manifests/</span> directory with sync wave 0.</span><br />
+<br />
+<h3 style='display: inline' id='challenge-3-secrets-management'>Challenge 3: Secrets Management</h3><br />
+<br />
+<span>ArgoCD stores Application manifests in Git, but secrets shouldn&#39;t be committed in plaintext.</span><br />
+<br />
+<span>**Solution (current)**: Secrets are created manually with <span class='inlinecode'>kubectl create secret</span> and referenced by the Helm charts. The secrets themselves aren&#39;t managed by ArgoCD.</span><br />
+<br />
+<span>**Future enhancement**: Migrate to External Secrets Operator (ESO) to manage secrets declaratively while storing the actual secrets in a separate backend (Kubernetes secrets in a separate namespace, or eventually Vault).</span><br />
+<br />
+<h3 style='display: inline' id='challenge-4-grafana-not-reloading-datasources'>Challenge 4: Grafana Not Reloading Datasources</h3><br />
+<br />
+<span>After updating the Grafana datasources ConfigMap, Grafana wouldn&#39;t detect the changes until pods were manually deleted.</span><br />
+<br />
+<span>**Solution**: Created a PostSync hook that automatically restarts Grafana pods after every ArgoCD sync. This runs as a Kubernetes Job in sync wave 10, ensuring it executes after all other resources are deployed.</span><br />
+<br />
+<h3 style='display: inline' id='challenge-5-prometheus-with-multiple-sources'>Challenge 5: Prometheus With Multiple Sources</h3><br />
+<br />
+<span>Prometheus needed both the upstream Helm chart and custom manifests (recording rules, dashboards, PVs).</span><br />
+<br />
+<span>**Solution**: Used ArgoCD&#39;s multi-source feature to combine:</span><br />
+<ul>
+<li>Helm chart from <span class='inlinecode'>prometheus-community.github.io/helm-charts</span></li>
+<li>Additional manifests from <span class='inlinecode'>codeberg.org/snonux/conf.git</span> at path <span class='inlinecode'>f3s/prometheus/manifests</span></li>
+</ul><br />
+<span>This keeps the upstream chart cleanly separated from custom configuration.</span><br />
+<br />
+<h3 style='display: inline' id='challenge-6-sync-ordering-for-prometheus'>Challenge 6: Sync Ordering for Prometheus</h3><br />
+<br />
+<span>Prometheus resources have dependencies:</span><br />
+<ul>
+<li>PVs before PVCs</li>
+<li>Secrets before Prometheus Operator</li>
+<li>PrometheusRule CRDs before Prometheus Operator can process them</li>
+<li>Grafana must be running before the restart hook executes</li>
+</ul><br />
+<span>**Solution**: Added sync wave annotations to all resources in <span class='inlinecode'>prometheus/manifests/</span>:</span><br />
+<ul>
+<li>Wave 0: PVs, RBAC</li>
+<li>Wave 1: Secrets, ConfigMaps</li>
+<li>Wave 3: PrometheusRule CRDs (recording rules)</li>
+<li>Wave 4: Dashboard ConfigMaps</li>
+<li>Wave 10: PostSync hook (Grafana restart)</li>
+</ul><br />
+<span>ArgoCD deploys resources in wave order, ensuring correct sequencing.</span><br />
+<br />
+<h2 style='display: inline' id='justfile-evolution'>Justfile Evolution</h2><br />
+<br />
+<span>The Justfiles evolved from deployment tools to utility scripts:</span><br />
+<br />
+<span>**Before (Helm deployment)**:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>install:
+    helm install miniflux ./helm-chart -n services
+
+upgrade:
+    helm upgrade miniflux ./helm-chart -n services
+
+uninstall:
+    helm uninstall miniflux -n services
+</pre>
+<br />
+<span>**After (ArgoCD utilities)**:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>status:
+    @kubectl get pods -n services -l app=miniflux
+    @kubectl get application miniflux -n cicd -o jsonpath=<font color="#808080">'Sync: {.status.sync.status}, Health: {.status.health.status}'</font>
+
+sync:
+    @kubectl annotate application miniflux -n cicd argocd.argoproj.io/refresh=normal --overwrite
+
+argocd-status:
+    argocd app get miniflux --core
+
+logs:
+    kubectl logs -n services -l app=miniflux --tail=<font color="#000000">100</font> -f
+</pre>
+<br />
+<span>The Justfiles now provide:</span><br />
+<ul>
+<li><span class='inlinecode'>status</span>: Quick health check</li>
+<li><span class='inlinecode'>sync</span>: Force immediate ArgoCD sync (instead of waiting 3 minutes)</li>
+<li><span class='inlinecode'>argocd-status</span>: Detailed ArgoCD application status</li>
+<li><span class='inlinecode'>logs</span>: Tail application logs</li>
+<li>Application-specific utilities (e.g., <span class='inlinecode'>port-forward</span>, <span class='inlinecode'>restart</span>)</li>
+</ul><br />
+<h2 style='display: inline' id='lessons-learned'>Lessons Learned</h2><br />
+<br />
+<span>1. **Incremental migration is safer than big-bang**: Migrating one app at a time allowed me to validate the pattern and fix issues before they affected all apps.</span><br />
+<br />
+<span>2. **Start with simple apps**: The first migration (simple services) established the basic pattern. Complex apps (Prometheus) came later after the pattern was proven.</span><br />
+<br />
+<span>3. **Sync waves are essential for complex apps**: Without sync waves, resources deployed in random order and caused failures. Proper ordering eliminated all deployment issues.</span><br />
+<br />
+<span>4. **Multi-source is powerful**: Combining upstream Helm charts with custom manifests keeps configuration clean and maintainable.</span><br />
+<br />
+<span>5. **PostSync hooks replace manual steps**: The Grafana restart hook eliminated a manual step that was easy to forget.</span><br />
+<br />
+<span>6. **Documentation in Git is better than tribal knowledge**: The Application manifests document exactly what&#39;s deployed and how. No more "let me check my shell history to remember how I deployed this."</span><br />
+<br />
+<span>7. **Self-healing prevents configuration drift**: Multiple times I&#39;ve manually tweaked something for debugging, forgotten about it, and ArgoCD automatically reverted it back to the desired state.</span><br />
+<br />
+<span>8. **ArgoCD Web UI is invaluable**: Seeing the resource tree, sync status, and health status at a glance is much better than running multiple <span class='inlinecode'>kubectl</span> commands.</span><br />
+<br />
+<h2 style='display: inline' id='future-improvements'>Future Improvements</h2><br />
+<br />
+<h3 style='display: inline' id='1-external-secrets-operator'>1. External Secrets Operator</h3><br />
+<br />
+<span>Currently, secrets are manually created with <span class='inlinecode'>kubectl create secret</span>. This works but isn&#39;t declarative. Plan:</span><br />
+<br />
+<ul>
+<li>Deploy External Secrets Operator (ESO)</li>
+<li>Store actual secrets in a Kubernetes Secret in a separate <span class='inlinecode'>secrets</span> namespace</li>
+<li>Create ExternalSecret CRDs that reference the backend secrets</li>
+<li>ArgoCD manages the ExternalSecret CRDs, ESO creates the actual Secrets</li>
+</ul><br />
+<span>This makes secrets declarative while keeping them out of Git.</span><br />
+<br />
+<h3 style='display: inline' id='2-applicationset-for-similar-apps'>2. ApplicationSet for Similar Apps</h3><br />
+<br />
+<span>Many apps have nearly identical Application manifests (miniflux, freshrss, wallabag, etc.). ArgoCD ApplicationSets can generate multiple Applications from a template:</span><br />
+<br />
+<pre>
+apiVersion: argoproj.io/v1alpha1
+kind: ApplicationSet
+metadata:
+  name: simple-services
+  namespace: cicd
+spec:
+  generators:
+    - list:
+        elements:
+          - app: miniflux
+          - app: freshrss
+          - app: wallabag
+  template:
+    metadata:
+      name: &#39;{{app}}&#39;
+    spec:
+      project: default
+      source:
+        repoURL: https://codeberg.org/snonux/conf.git
+        targetRevision: master
+        path: &#39;f3s/{{app}}/helm-chart&#39;
+      destination:
+        server: https://kubernetes.default.svc
+        namespace: services
+      syncPolicy:
+        automated:
+          prune: true
+          selfHeal: true
+</pre>
+<br />
+<span>One ApplicationSet could replace 10+ individual Application manifests.</span><br />
+<br />
+<h3 style='display: inline' id='3-app-of-apps-pattern'>3. App-of-Apps Pattern</h3><br />
+<br />
+<span>Currently, all Application manifests are applied manually with <span class='inlinecode'>kubectl apply -f argocd-apps/ -R</span>. An alternative is the "app-of-apps" pattern:</span><br />
+<br />
+<span>Create a root Application that deploys all other Applications. With the namespace-organized structure, this could be done per-namespace or for the entire cluster:</span><br />
+<br />
+<pre>
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: root
+  namespace: cicd
+spec:
+  source:
+    repoURL: https://codeberg.org/snonux/conf.git
+    targetRevision: master
+    path: f3s/argocd-apps
+    directory:
+      recurse: true  # Recursively find all manifests in subdirectories
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: cicd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+</pre>
+<br />
+<span>Or create separate root apps per namespace:</span><br />
+<br />
+<pre>
+# root-monitoring.yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: root-monitoring
+  namespace: cicd
+spec:
+  source:
+    repoURL: https://codeberg.org/snonux/conf.git
+    targetRevision: master
+    path: f3s/argocd-apps/monitoring
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: cicd
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+</pre>
+<br />
+<span>Then disaster recovery becomes:</span><br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl apply -f root-app.yaml
+<i><font color="silver"># Root app deploys all 21 applications automatically</font></i>
+
+<i><font color="silver"># Or apply by namespace</font></i>
+$ kubectl apply -f root-monitoring.yaml
+$ kubectl apply -f root-services.yaml
+$ kubectl apply -f root-infra.yaml
+</pre>
+<br />
+<h3 style='display: inline' id='4-argocd-image-updater'>4. ArgoCD Image Updater</h3><br />
+<br />
+<span>For applications with custom Docker images (like the registry, tracing-demo), ArgoCD Image Updater can automatically update the image tag in Git when a new image is pushed:</span><br />
+<br />
+<pre>
+metadata:
+  annotations:
+    argocd-image-updater.argoproj.io/image-list: |
+      app=registry.f3s.buetow.org/miniflux:~^v
+    argocd-image-updater.argoproj.io/write-back-method: git
+</pre>
+<br />
+<span>When a new image <span class='inlinecode'>registry.f3s.buetow.org/miniflux:v2.1.0</span> is pushed, Image Updater automatically:</span><br />
+<span>1. Updates the Helm values in Git</span><br />
+<span>2. Commits the change</span><br />
+<span>3. ArgoCD syncs the new image</span><br />
+<br />
+<span>This creates a fully automated CI/CD pipeline.</span><br />
+<br />
+<h2 style='display: inline' id='summary'>Summary</h2><br />
+<br />
+<span>Migrating from imperative Helm deployments to declarative GitOps with ArgoCD transformed how I manage the f3s cluster:</span><br />
+<br />
+<span>**Before**:</span><br />
+<ul>
+<li>Manual Helm commands for every change</li>
+<li>No visibility into cluster state</li>
+<li>Difficult to track what changed and when</li>
+<li>Disaster recovery required rebuilding from memory/notes</li>
+</ul><br />
+<span>**After**:</span><br />
+<ul>
+<li>Git is the single source of truth</li>
+<li>Automatic synchronization of changes</li>
+<li>Complete audit trail in Git history</li>
+<li>Drift detection and self-healing</li>
+<li>Disaster recovery: deploy ArgoCD, apply Application manifests, done</li>
+<li>Organized by namespace for clarity</li>
+</ul><br />
+<span>The migration took several days spread over a few weeks, migrating one application at a time. The result is a more maintainable, reliable, and recoverable cluster.</span><br />
+<br />
+<span>All 21 applications are now managed via GitOps, with the configuration living in:</span><br />
+<br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
+<br />
+<span>The ArgoCD Application manifests are organized by namespace:</span><br />
+<br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/argocd-apps'>codeberg.org/snonux/conf/f3s/argocd-apps</a><br />
+<br />
+<span>ArgoCD has become an essential part of the f3s infrastructure, and I can&#39;t imagine managing the cluster without it.</span><br />
+<br />
+<span>Other *BSD-related posts:</span><br />
+<br />
+<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br />
+<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br />
+<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
+<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
+<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
+<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
+<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
+<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
+<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
+<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
+<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
+<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let&#39;s Encrypt with OpenBSD and Rex</a><br />
+<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
+<br />
+<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span></span><br />
+<br />
+<a class='textlink' href='../'>Back to the main site</a><br />
+<p class="footer">
+	Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
+	served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
+	<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
+	<br />
+	Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a>
+</p>
+</body>
+</html>
diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml
index 3a5d9294..ed0ef983 100644
--- a/gemfeed/atom.xml
+++ b/gemfeed/atom.xml
@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <feed xmlns="http://www.w3.org/2005/Atom">
-    <updated>2026-01-07T23:43:40+02:00</updated>
+    <updated>2026-01-07T23:57:53+02:00</updated>
     <title>foo.zone feed</title>
     <subtitle>To be in the .zone!</subtitle>
     <link href="https://foo.zone/gemfeed/atom.xml" rel="self" />
@@ -2312,6 +2312,7 @@ $ curl -s -G "http://localhost:3200/api/search" \
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-8-observability'>f3s: Kubernetes with FreeBSD - Part 8: Observability</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#persistent-storage-recap'>Persistent storage recap</a></li>
 <li>⇢ <a href='#the-monitoring-namespace'>The monitoring namespace</a></li>
 <li>⇢ <a href='#installing-prometheus-and-grafana'>Installing Prometheus and Grafana</a></li>
@@ -2353,7 +2354,30 @@ $ curl -s -G "http://localhost:3200/api/search" \
 <br />
 <span>All manifests for the f3s stack live in my configuration repository:</span><br />
 <br />
-<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s </a><br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
+<br />
+<h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
+<br />
+<span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
+<br />
+<span>**To view the exact configuration as it existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
+$ cd f3s/prometheus/
+</pre>
+<br />
+<span>**Current master branch** contains the ArgoCD-managed versions with:</span><br />
+<span>- Application manifests organized under <span class='inlinecode'>argocd-apps/{monitoring,services,infra,test}/</span></span><br />
+<span>- Resources organized under <span class='inlinecode'>prometheus/manifests/</span>, <span class='inlinecode'>loki/</span>, etc.</span><br />
+<span>- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands</span><br />
+<br />
+<span>The deployment concepts and architecture remain the same—only the deployment method changed from imperative (<span class='inlinecode'>helm install/upgrade</span>) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series. </span><br />
 <br />
 <h2 style='display: inline' id='persistent-storage-recap'>Persistent storage recap</h2><br />
 <br />
@@ -2386,20 +2410,6 @@ namespace/monitoring created
 <br />
 <h2 style='display: inline' id='installing-prometheus-and-grafana'>Installing Prometheus and Grafana</h2><br />
 <br />
-<span>**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for declarative deployment. To view the exact configuration as it existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
-$ cd f3s/prometheus/
-</pre>
-<br />
-<span>The current master branch contains the ArgoCD-managed versions with Application manifests under <span class='inlinecode'>argocd-apps/</span> and resources organized under <span class='inlinecode'>prometheus/manifests/</span>, <span class='inlinecode'>loki/</span>, etc. The Justfiles have been updated to trigger ArgoCD syncs instead of direct Helm commands.</span><br />
-<br />
 <span>Prometheus and Grafana are deployed together using the <span class='inlinecode'>kube-prometheus-stack</span> Helm chart from the Prometheus community. This chart bundles Prometheus, Grafana, Alertmanager, and various exporters (Node Exporter, Kube State Metrics) into a single deployment. Ill explain what each component does in detail later when we look at the running pods.</span><br />
 <br />
 <h3 style='display: inline' id='prerequisites'>Prerequisites</h3><br />
@@ -3977,6 +3987,7 @@ p hash.values_at(:a, :c)
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-7-k3s-and-first-pod-deployments'>f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#updating'>Updating</a></li>
 <li>⇢ <a href='#installing-k3s'>Installing k3s</a></li>
 <li>⇢ ⇢ <a href='#generating-k3stoken-and-starting-the-first-k3s-node'>Generating <span class='inlinecode'>K3S_TOKEN</span> and starting the first k3s node</a></li>
@@ -4007,6 +4018,29 @@ p hash.values_at(:a, :c)
 <br />
 <a class='textlink' href='https://k3s.io'>https://k3s.io</a><br />
 <br />
+<h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
+<br />
+<span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests and Helm charts in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
+<br />
+<span>**To view the exact manifests and charts as they existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
+$ cd f3s/
+</pre>
+<br />
+<span>**Current master branch** contains the ArgoCD-managed versions with:</span><br />
+<span>- Application manifests organized under <span class='inlinecode'>argocd-apps/{monitoring,services,infra,test}/</span></span><br />
+<span>- Additional resources under <span class='inlinecode'>*/manifests/</span> directories (e.g., <span class='inlinecode'>prometheus/manifests/</span>)</span><br />
+<span>- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands</span><br />
+<br />
+<span>The deployment concepts and architecture remain the same—only the deployment method changed from imperative (<span class='inlinecode'>helm install/upgrade</span>) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series.</span><br />
+<br />
 <h2 style='display: inline' id='updating'>Updating</h2><br />
 <br />
 <span>Before proceeding, I bring all systems involved up-to-date. On all three Rocky Linux 9 boxes <span class='inlinecode'>r0</span>, <span class='inlinecode'>r1</span>, and <span class='inlinecode'>r2</span>:</span><br />
@@ -4822,21 +4856,7 @@ http://www.gnu.org/software/src-highlite -->
 <br />
 <a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
 <br />
-<span>**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests and Helm charts in the repository have been reorganized for declarative deployment. To view the exact manifests and charts as they existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
-$ cd f3s/
-</pre>
-<br />
-<span>The current master branch contains the ArgoCD-managed versions with manifests organized under <span class='inlinecode'>argocd-apps/</span> and <span class='inlinecode'>*/manifests/</span> directories.</span><br />
-<br />
-<span>Within that repo, the <span class='inlinecode'>examples/conf/f3s/registry/</span> directory contains the Helm chart, a <span class='inlinecode'>Justfile</span>, and a detailed <span class='inlinecode'>README</span>. Here&#39;s the condensed walkthrough I used to roll out the registry with Helm.</span><br />
+<span>Within that repo, the <span class='inlinecode'>f3s/registry/</span> directory contains the Helm chart, a <span class='inlinecode'>Justfile</span>, and a detailed <span class='inlinecode'>README</span>. Here&#39;s the condensed walkthrough I used to roll out the registry with Helm.</span><br />
 <br />
 <h3 style='display: inline' id='prepare-the-nfs-backed-storage'>Prepare the NFS-backed storage</h3><br />
 <br />
diff --git a/index.html b/index.html
index cdd66390..1071568f 100644
--- a/index.html
+++ b/index.html
@@ -13,7 +13,7 @@
 </p>
 <h1 style='display: inline' id='hello'>Hello!</h1><br />
 <br />
-<span class='quote'>This site was generated at 2026-01-07T23:43:40+02:00 by <span class='inlinecode'>Gemtexter</span></span><br />
+<span class='quote'>This site was generated at 2026-01-07T23:57:53+02:00 by <span class='inlinecode'>Gemtexter</span></span><br />
 <br />
 <span>Welcome to the foo.zone!</span><br />
 <br />
diff --git a/uptime-stats.html b/uptime-stats.html
index 11529764..794ee1bc 100644
--- a/uptime-stats.html
+++ b/uptime-stats.html
@@ -13,7 +13,7 @@
 </p>
 <h1 style='display: inline' id='my-machine-uptime-stats'>My machine uptime stats</h1><br />
 <br />
-<span class='quote'>This site was last updated at 2026-01-07T23:43:40+02:00</span><br />
+<span class='quote'>This site was last updated at 2026-01-07T23:57:53+02:00</span><br />
 <br />
 <span>The following stats were collected via <span class='inlinecode'>uptimed</span> on all of my personal computers over many years and the output was generated by <span class='inlinecode'>guprecords</span>, the global uptime records stats analyser of mine.</span><br />
 <br />