summaryrefslogtreecommitdiff
path: root/gemfeed
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2025-07-09 00:54:07 +0300
committerPaul Buetow <paul@buetow.org>2025-07-09 00:54:07 +0300
commit24636ef8fc5fc11e878a11fb8ae821d8e7d0a269 (patch)
tree1094e8ead64ecaed2da77c126a1386bcfd61d68c /gemfeed
parentcbc7ebcde06e453235ab5ea1b66f754c560e3a7f (diff)
Update content for md
Diffstat (limited to 'gemfeed')
-rw-r--r--gemfeed/2025-01-01-posts-from-october-to-december-2024.md1
-rw-r--r--gemfeed/2025-06-22-task-samurai.md2
-rw-r--r--gemfeed/2025-07-01-posts-from-january-to-june-2025.md731
-rw-r--r--gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.md2536
-rwxr-xr-xgemfeed/configure-stunnel-nfs-r1-r2.sh163
-rw-r--r--gemfeed/index.md1
-rw-r--r--gemfeed/stunnel-nfs-quick-reference.txt78
7 files changed, 3487 insertions, 25 deletions
diff --git a/gemfeed/2025-01-01-posts-from-october-to-december-2024.md b/gemfeed/2025-01-01-posts-from-october-to-december-2024.md
index 0bdc34d7..a44041ee 100644
--- a/gemfeed/2025-01-01-posts-from-october-to-december-2024.md
+++ b/gemfeed/2025-01-01-posts-from-october-to-december-2024.md
@@ -327,6 +327,7 @@ My New Year's resolution is not to start any new non-fiction books (or only very
Other related posts:
+[2025-07-01 Posts from January to June 2025](./2025-07-01-posts-from-january-to-june-2025.md)
[2025-01-01 Posts from October to December 2024 (You are currently reading this)](./2025-01-01-posts-from-october-to-december-2024.md)
E-Mail your comments to `paul@nospam.buetow.org` :-)
diff --git a/gemfeed/2025-06-22-task-samurai.md b/gemfeed/2025-06-22-task-samurai.md
index a647ef81..b6483478 100644
--- a/gemfeed/2025-06-22-task-samurai.md
+++ b/gemfeed/2025-06-22-task-samurai.md
@@ -29,7 +29,7 @@ Task Samurai is a fast terminal interface for Taskwarrior written in Go using th
### Why does this exist?
-* I wanted to tinker with agentic coding. This project was entirely implemented using OpenAI Codex.
+I wanted to tinker with agentic coding. This project was implemented entirely using OpenAI Codex. (After this blog post was published, I also used the Claude Code CLI.)
* I wanted a faster UI for Taskwarrior than other options, like Vit, which is Python-based.
* I wanted something built with Bubble Tea, but I never had time to dive deep into it.
* I wanted to build a toy project (like Task Samurai) first, before tackling the big ones, to get started with agentic coding.
diff --git a/gemfeed/2025-07-01-posts-from-january-to-june-2025.md b/gemfeed/2025-07-01-posts-from-january-to-june-2025.md
new file mode 100644
index 00000000..0620a68b
--- /dev/null
+++ b/gemfeed/2025-07-01-posts-from-january-to-june-2025.md
@@ -0,0 +1,731 @@
+# Posts from January to June 2025
+
+> Published at 2025-07-01T22:39:29+03:00
+
+These are my social media posts from the last six months. I keep them here to reflect on them and also to not lose them. Social media networks come and go and are not under my control, but my domain is here to stay.
+
+These are from Mastodon and LinkedIn. Have a look at my about page for my social media profiles. This list is generated with Gos, my social media platform sharing tool.
+
+[My about page](../about/index.md)
+[https://codeberg.org/snonux/gos](https://codeberg.org/snonux/gos)
+
+## Table of Contents
+
+* [⇢ Posts from January to June 2025](#posts-from-january-to-june-2025)
+* [⇢ ⇢ January 2025](#january-2025)
+* [⇢ ⇢ ⇢ I am currently binge-listening to the Google ...](#i-am-currently-binge-listening-to-the-google-)
+* [⇢ ⇢ ⇢ Recently, there was a >5000 LOC `#bash` ...](#recently-there-was-a-5000-loc-bash-)
+* [⇢ ⇢ ⇢ Ghostty is a terminal emulator that was ...](#ghostty-is-a-terminal-emulator-that-was-)
+* [⇢ ⇢ ⇢ Go is not an easy programming language. Don't ...](#go-is-not-an-easy-programming-language-don-t-)
+* [⇢ ⇢ ⇢ How will AI change software engineering (or has ...](#how-will-ai-change-software-engineering-or-has-)
+* [⇢ ⇢ ⇢ Eliminating toil - Toil is not always a bad ...](#eliminating-toil---toil-is-not-always-a-bad-)
+* [⇢ ⇢ ⇢ Fun read. How about using the character ...](#fun-read-how-about-using-the-character-)
+* [⇢ ⇢ ⇢ Thats unexpected, you cant remove a NaN key ...](#thats-unexpected-you-cant-remove-a-nan-key-)
+* [⇢ ⇢ ⇢ Nice refresher for `#shell` `#bash` `#zsh` ...](#nice-refresher-for-shell-bash-zsh-)
+* [⇢ ⇢ ⇢ I think discussing action items in incident ...](#i-think-discussing-action-items-in-incident-)
+* [⇢ ⇢ ⇢ At first, functional options add a bit of ...](#at-first-functional-options-add-a-bit-of-)
+* [⇢ ⇢ ⇢ In the "Working with an SRE Interview" I have ...](#in-the-working-with-an-sre-interview-i-have-)
+* [⇢ ⇢ ⇢ Small introduction to the `#Android` ...](#small-introduction-to-the-android-)
+* [⇢ ⇢ ⇢ Helix 2025.01 has been released. The completion ...](#helix-202501-has-been-released-the-completion-)
+* [⇢ ⇢ ⇢ I found these are excellent examples of how ...](#i-found-these-are-excellent-examples-of-how-)
+* [⇢ ⇢ ⇢ LLMs for Ops? Summaries of logs, probabilities ...](#llms-for-ops-summaries-of-logs-probabilities-)
+* [⇢ ⇢ ⇢ Enjoying an APC Power-UPS BX750MI in my ...](#enjoying-an-apc-power-ups-bx750mi-in-my-)
+* [⇢ ⇢ ⇢ "Even in the projects where I'm the only ...](#even-in-the-projects-where-i-m-the-only-)
+* [⇢ ⇢ ⇢ Connecting an `#UPS` to my `#FreeBSD` cluster ...](#connecting-an-ups-to-my-freebsd-cluster-)
+* [⇢ ⇢ ⇢ So, the Co-founder and CTO of honeycomb.io and ...](#so-the-co-founder-and-cto-of-honeycombio-and-)
+* [⇢ ⇢ February 2025](#february-2025)
+* [⇢ ⇢ ⇢ I don't know about you, but at work, I usually ...](#i-don-t-know-about-you-but-at-work-i-usually-)
+* [⇢ ⇢ ⇢ Great proposal (got accepted by the Goteam) for ...](#great-proposal-got-accepted-by-the-goteam-for-)
+* [⇢ ⇢ ⇢ My Gemtexter has only 1320 LOC.... The Biggest ...](#my-gemtexter-has-only-1320-loc-the-biggest-)
+* [⇢ ⇢ ⇢ Against /tmp - He is making a point `#unix` ...](#against-tmp---he-is-making-a-point-unix-)
+* [⇢ ⇢ ⇢ Random Weird Things Part 2: `#blog` ...](#random-weird-things-part-2-blog-)
+* [⇢ ⇢ ⇢ As a former `#Pebble` user and fan, thats ...](#as-a-former-pebble-user-and-fan-thats-)
+* [⇢ ⇢ ⇢ I think I am slowly getting the point of Cue. ...](#i-think-i-am-slowly-getting-the-point-of-cue-)
+* [⇢ ⇢ ⇢ Jonathan's reflection of 10 years of ...](#jonathan-s-reflection-of-10-years-of-)
+* [⇢ ⇢ ⇢ Really enjoyed reading this. Easily digestible ...](#really-enjoyed-reading-this-easily-digestible-)
+* [⇢ ⇢ ⇢ Some great advice from 40 years of experience ...](#some-great-advice-from-40-years-of-experience-)
+* [⇢ ⇢ ⇢ I enjoyed this talk, some recipes I knew ...](#i-enjoyed-this-talk-some-recipes-i-knew-)
+* [⇢ ⇢ ⇢ A way of how to add the version info to the Go ...](#a-way-of-how-to-add-the-version-info-to-the-go-)
+* [⇢ ⇢ ⇢ In other words, using t.Parallel() for ...](#in-other-words-using-tparallel-for-)
+* [⇢ ⇢ ⇢ Neat little blog post, showcasing various ...](#neat-little-blog-post-showcasing-various-)
+* [⇢ ⇢ ⇢ The smallest thing in Go `#golang` ...](#the-smallest-thing-in-go-golang-)
+* [⇢ ⇢ ⇢ Fun with defer in `#golang`, I did't know, that ...](#fun-with-defer-in-golang-i-did-t-know-that-)
+* [⇢ ⇢ ⇢ What I like about Go is that it is still ...](#what-i-like-about-go-is-that-it-is-still-)
+* [⇢ ⇢ March 2025](#march-2025)
+* [⇢ ⇢ ⇢ Television has somewhat transformed how I work ...](#television-has-somewhat-transformed-how-i-work-)
+* [⇢ ⇢ ⇢ Once in a while, I like to read a book about a ...](#once-in-a-while-i-like-to-read-a-book-about-a-)
+* [⇢ ⇢ ⇢ As you may have noticed, I like to share on ...](#as-you-may-have-noticed-i-like-to-share-on-)
+* [⇢ ⇢ ⇢ Personally, I think AI (LLMs) are pretty ...](#personally-i-think-ai-llms-are-pretty-)
+* [⇢ ⇢ ⇢ Type aliases in `#golang`, soon also work with ...](#type-aliases-in-golang-soon-also-work-with-)
+* [⇢ ⇢ ⇢ `#Perl`, my "first love" of programming ...](#perl-my-first-love-of-programming-)
+* [⇢ ⇢ ⇢ I guess there are valid reasons for phttpdget, ...](#i-guess-there-are-valid-reasons-for-phttpdget-)
+* [⇢ ⇢ ⇢ This is one of the reasons why I like ...](#this-is-one-of-the-reasons-why-i-like-)
+* [⇢ ⇢ ⇢ Advanced Concurrency Patterns with `#Golang` ...](#advanced-concurrency-patterns-with-golang-)
+* [⇢ ⇢ ⇢ `#SQLite` was designed as an `#TCL` extension. ...](#sqlite-was-designed-as-an-tcl-extension-)
+* [⇢ ⇢ ⇢ Git provides automatic rendering of Markdown ...](#git-provides-automatic-rendering-of-markdown-)
+* [⇢ ⇢ ⇢ These are some neat little Go tips. Linters ...](#these-are-some-neat-little-go-tips-linters-)
+* [⇢ ⇢ ⇢ This is a great introductory blog post about ...](#this-is-a-great-introductory-blog-post-about-)
+* [⇢ ⇢ ⇢ Maps in Go under the hood `#golang` ...](#maps-in-go-under-the-hood-golang-)
+* [⇢ ⇢ ⇢ I found that working on multiple side projects ...](#i-found-that-working-on-multiple-side-projects-)
+* [⇢ ⇢ ⇢ I have been in incidents. Understandably, ...](#i-have-been-in-incidents-understandably-)
+* [⇢ ⇢ ⇢ I dont understand what it is. Certificates are ...](#i-dont-understand-what-it-is-certificates-are-)
+* [⇢ ⇢ ⇢ Don't just blindly trust LLMs. I recently ...](#don-t-just-blindly-trust-llms-i-recently-)
+* [⇢ ⇢ April 2025](#april-2025)
+* [⇢ ⇢ ⇢ I knew about any being equivalent to ...](#i-knew-about-any-being-equivalent-to-)
+* [⇢ ⇢ ⇢ Neat summary of new `#Perl` features per ...](#neat-summary-of-new-perl-features-per-)
+* [⇢ ⇢ ⇢ errors.As() checks for the error type, whereas ...](#errorsas-checks-for-the-error-type-whereas-)
+* [⇢ ⇢ ⇢ Good stuff: 10 years of functional options and ...](#good-stuff-10-years-of-functional-options-and-)
+* [⇢ ⇢ ⇢ I had some fun with `#FreeBSD`, `#Bhyve` and ...](#i-had-some-fun-with-freebsd-bhyve-and-)
+* [⇢ ⇢ ⇢ The moment your blog receives PRs for typo ...](#the-moment-your-blog-receives-prs-for-typo-)
+* [⇢ ⇢ ⇢ One thing not mentioned is that `#OpenRsync`'s ...](#one-thing-not-mentioned-is-that-openrsync-s-)
+* [⇢ ⇢ ⇢ This is an interesting `#Elixir` pipes operator ...](#this-is-an-interesting-elixir-pipes-operator-)
+* [⇢ ⇢ ⇢ The story of how my favorite `#Golang` book was ...](#the-story-of-how-my-favorite-golang-book-was-)
+* [⇢ ⇢ ⇢ These are my personal book notes from Daniel ...](#these-are-my-personal-book-notes-from-daniel-)
+* [⇢ ⇢ ⇢ I certainly learned a lot reading this `#llm` ...](#i-certainly-learned-a-lot-reading-this-llm-)
+* [⇢ ⇢ ⇢ Writing indempotent `#Bash` scripts ...](#writing-indempotent-bash-scripts-)
+* [⇢ ⇢ ⇢ Regarding `#AI` for code generation. You should ...](#regarding-ai-for-code-generation-you-should-)
+* [⇢ ⇢ ⇢ I like the Rocky metaphor. And this post also ...](#i-like-the-rocky-metaphor-and-this-post-also-)
+* [⇢ ⇢ May 2025](#may-2025)
+* [⇢ ⇢ ⇢ There's now also a `#Fish` shell edition of my ...](#there-s-now-also-a-fish-shell-edition-of-my-)
+* [⇢ ⇢ ⇢ I loved this talk. It's about how you can ...](#i-loved-this-talk-it-s-about-how-you-can-)
+* [⇢ ⇢ ⇢ Some unexpected `#golang` stuff, ppl say, that ...](#some-unexpected-golang-stuff-ppl-say-that-)
+* [⇢ ⇢ ⇢ With the advent of AI and LLMs, I have observed ...](#with-the-advent-of-ai-and-llms-i-have-observed-)
+* [⇢ ⇢ ⇢ For science, fun and profit, I set up a ...](#for-science-fun-and-profit-i-set-up-a-)
+* [⇢ ⇢ ⇢ Ever wondered about the hung task Linux ...](#ever-wondered-about-the-hung-task-linux-)
+* [⇢ ⇢ ⇢ A bit of `#fun`: The FORTRAN hating gateway ― ...](#a-bit-of-fun-the-fortran-hating-gateway--)
+* [⇢ ⇢ ⇢ So, Golang was invented while engineers at ...](#so-golang-was-invented-while-engineers-at-)
+* [⇢ ⇢ ⇢ I couldn't do without here-docs. If they did ...](#i-couldn-t-do-without-here-docs-if-they-did-)
+* [⇢ ⇢ ⇢ I started using computers as a kid on MS-DOS ...](#i-started-using-computers-as-a-kid-on-ms-dos-)
+* [⇢ ⇢ ⇢ Thats interesting, running `#Android` in ...](#thats-interesting-running-android-in-)
+* [⇢ ⇢ ⇢ Before wiping the pre-installed `#Windows` 11 ...](#before-wiping-the-pre-installed-windows-11-)
+* [⇢ ⇢ ⇢ Some might hate me saying this, but didnt ...](#some-might-hate-me-saying-this-but-didnt-)
+* [⇢ ⇢ ⇢ Wouldn't still do that, even with 100% test ...](#wouldn-t-still-do-that-even-with-100-test-)
+* [⇢ ⇢ ⇢ Some neat slice tricks for Go: `#golang` ...](#some-neat-slice-tricks-for-go-golang-)
+* [⇢ ⇢ ⇢ I understand that Kubernetes is not for ...](#i-understand-that-kubernetes-is-not-for-)
+* [⇢ ⇢ June 2025](#june-2025)
+* [⇢ ⇢ ⇢ Some great advices, will try out some of them! ...](#some-great-advices-will-try-out-some-of-them-)
+* [⇢ ⇢ ⇢ In `#Golang`, values are actually copied when ...](#in-golang-values-are-actually-copied-when-)
+* [⇢ ⇢ ⇢ This is a great little tutorial for searching ...](#this-is-a-great-little-tutorial-for-searching-)
+* [⇢ ⇢ ⇢ The mov instruction of a CPU is turing ...](#the-mov-instruction-of-a-cpu-is-turing-)
+* [⇢ ⇢ ⇢ I removed the social media profile from my ...](#i-removed-the-social-media-profile-from-my-)
+* [⇢ ⇢ ⇢ So want a "real" recent UNIX? Use AIX! `#macos` ...](#so-want-a-real-recent-unix-use-aix-macos-)
+* [⇢ ⇢ ⇢ This episode, I think, is kind of an eye-opener ...](#this-episode-i-think-is-kind-of-an-eye-opener-)
+* [⇢ ⇢ ⇢ My `#OpenBSD` blog setup got mentioned in the ...](#my-openbsd-blog-setup-got-mentioned-in-the-)
+* [⇢ ⇢ ⇢ `#Golang` is the best when it comes to agentic ...](#golang-is-the-best-when-it-comes-to-agentic-)
+* [⇢ ⇢ ⇢ Where `#zsh` is better than `#bash` ...](#where-zsh-is-better-than-bash-)
+* [⇢ ⇢ ⇢ I really enjoyed this talk about obscure Go ...](#i-really-enjoyed-this-talk-about-obscure-go-)
+* [⇢ ⇢ ⇢ Commenting your regular expression is generally ...](#commenting-your-regular-expression-is-generally-)
+* [⇢ ⇢ ⇢ You have to make a decision for yourself, but ...](#you-have-to-make-a-decision-for-yourself-but-)
+* [⇢ ⇢ ⇢ "100 Go Mistakes and How to Avoid Them" is one ...](#100-go-mistakes-and-how-to-avoid-them-is-one-)
+* [⇢ ⇢ ⇢ The `#Ruby` Data class seems quite helpful ...](#the-ruby-data-class-seems-quite-helpful-)
+
+## January 2025
+
+### I am currently binge-listening to the Google ...
+
+I am currently binge-listening to the Google `#SRE` ProdCast. It's really great to learn about the stories of individual SREs and their journeys. It is not just about SREs at Google; there are also external guests.
+
+[sre.google/prodcast/](https://sre.google/prodcast/)
+
+### Recently, there was a >5000 LOC `#bash` ...
+
+Recently, there was a >5000 LOC `#bash` codebase at work that reported the progress of a migration, nobody understood it and it was wonky (sometimes it would not return the desired results). On top of that, the coding style was very bad as well (I could rant forever here). The engineer who wrote it left the company. I rewrote it in `#Perl` in about 300 LOC. Colleagues asked why not Python. Perl is the perfect choice here—it's even in its name: Practical Extraction and Report Language!
+
+### Ghostty is a terminal emulator that was ...
+
+Ghostty is a terminal emulator that was recently released publicly as open-source. I love that it works natively on both Linux and macOS; it looks great (font rendering) and is fast and customizable via a config file (which I manage with a config mng system). Ghostty is a passion project written in Zig, the author loved the community so much while working on it that he donated $300k to the Zig Foundation. `#terminal` `#emulator`
+
+[ghostty.org](https://ghostty.org)
+
+### Go is not an easy programming language. Don't ...
+
+Go is not an easy programming language. Don't confuse easy with simple syntax. I'd agree to this. With the recent addition of Generics to the language I also feel that even the syntax stops being simple.. Also, simplicity is complex (especially under the hood how the language works - there are many mechanics you need to know if you really want to master the language). `#golang`
+
+[www.arp242.net/go-easy.html](https://www.arp242.net/go-easy.html)
+
+### How will AI change software engineering (or has ...
+
+How will AI change software engineering (or has it already)? The bottom line is that less experienced engineers may have problems (accepting incomplete or incorrect programs, only reaching 70 percent solutions), while experienced engineers can leverage AI to boost their performance as they know how to fix the remaining 30 percent of the generated code. `#ai` `#engineering` `#software`
+
+[newsletter.pragmaticengineer.com/p/how-ai-will-change-software-engineering](https://newsletter.pragmaticengineer.com/p/how-ai-will-change-software-engineering)
+
+### Eliminating toil - Toil is not always a bad ...
+
+Eliminating toil - Toil is not always a bad thing - some even enjoy toil - it is calming in small amounts - but it becomes toxic in large amounts - `#SRE`
+
+[sre.google/sre-book/eliminating-toil/](https://sre.google/sre-book/eliminating-toil/)
+
+### Fun read. How about using the character ...
+
+Fun read. How about using the character sequence :-) as a statement separator in a programming language?
+
+[ntietz.com/blog/researching-why-we-use-semicolons-as-statement-terminators/](https://ntietz.com/blog/researching-why-we-use-semicolons-as-statement-terminators/)
+
+### Thats unexpected, you cant remove a NaN key ...
+
+Thats unexpected, you cant remove a NaN key from a map without clearing it! `#golang` via @wallabagapp
+
+[unexpected-go.com/you-cant-remove-a-nan-key-from-a-map-without-clearing-it.html](https://unexpected-go.com/you-cant-remove-a-nan-key-from-a-map-without-clearing-it.html)
+
+### Nice refresher for `#shell` `#bash` `#zsh` ...
+
+Nice refresher for `#shell` `#bash` `#zsh` redirection rules
+
+[rednafi.com/misc/shell_redirection/](https://rednafi.com/misc/shell_redirection/)
+
+### I think discussing action items in incident ...
+
+I think discussing action items in incident reviews is important. At least the obvious should be captured and noted down. It does not mean that the action items need to be fully refined in the review meeting; that would be out of scope, in my opinion.
+
+[surfingcomplexity.blog/2024/09/28/why-..-..-action-items-during-incident-reviews/](https://surfingcomplexity.blog/2024/09/28/why-i-dont-like-discussing-action-items-during-incident-reviews/)
+
+### At first, functional options add a bit of ...
+
+At first, functional options add a bit of boilerplate, but they turn out to be quite neat, especially when you have very long parameter lists that need to be made neat and tidy. `#golang`
+
+[www.calhoun.io/using-functional-options-instead-of-method-chaining-in-go/](https://www.calhoun.io/using-functional-options-instead-of-method-chaining-in-go/)
+
+### In the "Working with an SRE Interview" I have ...
+
+In the "Working with an SRE Interview" I have been askd about what it's like working with an SRE! We'd covered much more in depth, but we decided not to make it too long in the final version! `#sre` `#interview`
+
+[foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.gmi)
+[foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html](https://foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html)
+
+### Small introduction to the `#Android` ...
+
+Small introduction to the `#Android` distribution called `#GrapheneOS` For myself, I am using a Pixel 7 Pro, which comes with "only" 5 years of support (not yet 7 years like the Pixel 8 and 9 series). I also wrote about GrapheneOS here once:
+
+[dataswamp.org/~solene/2025-01-12-intro-to-grapheneos.html](https://dataswamp.org/~solene/2025-01-12-intro-to-grapheneos.html)
+[foo.zone/gemfeed/2023-01-23-why-grapheneos-rox.gmi (Gemini)](gemini://foo.zone/gemfeed/2023-01-23-why-grapheneos-rox.gmi)
+[foo.zone/gemfeed/2023-01-23-why-grapheneos-rox.html](https://foo.zone/gemfeed/2023-01-23-why-grapheneos-rox.html)
+
+### Helix 2025.01 has been released. The completion ...
+
+Helix 2025.01 has been released. The completion of path names and the snippet functionality will be particularly useful for me. Overall, it's a great release. The release notes cover only some highlights, but there are many more changes in this version so also have a look at the Changelog! `#HelixEditor`
+
+[helix-editor.com/news/release-25-01-highlights/](https://helix-editor.com/news/release-25-01-highlights/)
+
+### I found these are excellent examples of how ...
+
+I found these are excellent examples of how `#OpenBSD`'s `#relayd` can be used.
+
+[www.tumfatig.net/2023/using-openbsd-relayd8-as-an-application-layer-gateway/](https://www.tumfatig.net/2023/using-openbsd-relayd8-as-an-application-layer-gateway/)
+
+### LLMs for Ops? Summaries of logs, probabilities ...
+
+LLMs for Ops? Summaries of logs, probabilities about correctness, auto-generating Ansible, some uses cases are there. Wouldn't trust it fully, though.
+
+[youtu.be/WodaffxVq-E?si=noY0egrfl5izCSQI](https://youtu.be/WodaffxVq-E?si=noY0egrfl5izCSQI)
+
+### Enjoying an APC Power-UPS BX750MI in my ...
+
+Enjoying an APC Power-UPS BX750MI in my `#homelab` with `#FreeBSD` and apcupsd. I can easily use the UPS status to auto-shutdown a cluster of FreeBSD machines on a power cut. One FreeBSD machine acts as the apcupsd master, connected via USB to the APC, while the remaining machines read the status remotely via the apcupsd network port from the master. However, it won't work when the master is down. `#APC` `#UPS`
+
+### "Even in the projects where I'm the only ...
+
+"Even in the projects where I'm the only person, there are at least three people involved: past me, present me, and future me." - Quote from `#software` `#programming`
+
+[liw.fi/40/#index1h1](https://liw.fi/40/#index1h1)
+
+### Connecting an `#UPS` to my `#FreeBSD` cluster ...
+
+Connecting an `#UPS` to my `#FreeBSD` cluster in my `#homelab`, protecting it from power cuts!
+
+[foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi)
+[foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html](https://foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html)
+
+### So, the Co-founder and CTO of honeycomb.io and ...
+
+So, the Co-founder and CTO of honeycomb.io and author of the book Observability Engineering always hated observability. And Distinguished Software Engineer and The Pragmatic Engineer host can't pronounce the word Observability. :-) No, jokes aside, I liked this podcast episode of The Pragmatic Engineer: Observability: the present and future, with Charity Majors `#sre` `#observability`
+
+[newsletter.pragmaticengineer.com/p/observability-the-present-and-future](https://newsletter.pragmaticengineer.com/p/observability-the-present-and-future)
+
+## February 2025
+
+### I don't know about you, but at work, I usually ...
+
+I don't know about you, but at work, I usually deal with complex setups involving thousands of servers and work in a complex hybrid microservices-based environment (cloud and on-prem), where homelabbing (as simple as described in my blog post) is really relaxing and recreative. So, I was homelabbing a bit again, securing my `#FreeBSD` cluster from power cuts. `#UPS` `#recreative`
+
+[foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi)
+[foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html](https://foo.zone/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html)
+
+### Great proposal (got accepted by the Goteam) for ...
+
+Great proposal (got accepted by the Goteam) for safer file system open functions `#golang`
+
+[github.com/golang/go/issues/67002](https://github.com/golang/go/issues/67002)
+
+### My Gemtexter has only 1320 LOC.... The Biggest ...
+
+My Gemtexter has only 1320 LOC.... The Biggest Shell Programs in the World are huuuge... `#shell` `#sh`
+
+[github.com/oils-for-unix/oils/wiki/The-Biggest-Shell-Programs-in-the-World](https://github.com/oils-for-unix/oils/wiki/The-Biggest-Shell-Programs-in-the-World)
+
+### Against /tmp - He is making a point `#unix` ...
+
+Against /tmp - He is making a point `#unix` `#linux` `#bsd` `#filesystem` via @wallabagapp
+
+[dotat.at/@/2024-10-22-tmp.html](https://dotat.at/@/2024-10-22-tmp.html)
+
+### Random Weird Things Part 2: `#blog` ...
+
+Random Weird Things Part 2: `#blog` `#computing`
+
+[foo.zone/gemfeed/2025-02-08-random-weird-things-ii.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-02-08-random-weird-things-ii.gmi)
+[foo.zone/gemfeed/2025-02-08-random-weird-things-ii.html](https://foo.zone/gemfeed/2025-02-08-random-weird-things-ii.html)
+
+### As a former `#Pebble` user and fan, thats ...
+
+As a former `#Pebble` user and fan, thats aweaome news. PebbleOS is now open source and there will aoon be a new watch. I don't know about you, but I will be the first getting one :-) `#foss`
+
+[ericmigi.com/blog/why-were-bringing-pebble-back](https://ericmigi.com/blog/why-were-bringing-pebble-back)
+
+### I think I am slowly getting the point of Cue. ...
+
+I think I am slowly getting the point of Cue. For example, it can replace both a JSON file and a JSON Schema. Furthermore, you can convert it from and into different formats (Cue, JSON, YAML, Go data types, ...), and you can nicely embed this into a Go project as well. `#cue` `#cuelang` `#golang` `#configuration`
+
+[cuelang.org](https://cuelang.org)
+
+### Jonathan's reflection of 10 years of ...
+
+Jonathan's reflection of 10 years of programming!
+
+[jonathan-frere.com/posts/10-years-of-programming/](https://jonathan-frere.com/posts/10-years-of-programming/)
+
+### Really enjoyed reading this. Easily digestible ...
+
+Really enjoyed reading this. Easily digestible summary of what's new in Go 1.24. `#golang`
+
+[antonz.org/go-1-24/](https://antonz.org/go-1-24/)
+
+### Some great advice from 40 years of experience ...
+
+Some great advice from 40 years of experience as a software developer. `#software` `#development`
+
+[liw.fi/40/#index1h1](https://liw.fi/40/#index1h1)
+
+### I enjoyed this talk, some recipes I knew ...
+
+I enjoyed this talk, some recipes I knew already, others were new to me. The "line of sight" is my favourite, which I always tend to follow. I also liked the example where the speaker simplified a "complex" nested functions into two not-nested-if-statements. `#golang`
+
+[www.youtube.com/watch?v=zdKHq9Xo4OY&list=WL&index=5](https://www.youtube.com/watch?v=zdKHq9Xo4OY&list=WL&index=5)
+
+### A way of how to add the version info to the Go ...
+
+A way of how to add the version info to the Go binary. ... I personally just hardcode the version number in version.go and update it there manually for each release. But with Go 1.24, I will try embedding it! `#golang`
+
+[jerrynsh.com/3-easy-ways-to-add-version-flag-in-go/](https://jerrynsh.com/3-easy-ways-to-add-version-flag-in-go/)
+
+### In other words, using t.Parallel() for ...
+
+In other words, using t.Parallel() for lightweight unit tests will likely make them slower.... `#golang`
+
+[threedots.tech/post/go-test-parallelism/](https://threedots.tech/post/go-test-parallelism/)
+
+### Neat little blog post, showcasing various ...
+
+Neat little blog post, showcasing various methods unsed for generic programming before of the introduction of generics. Only reflection wasn't listed. `#golang`
+
+[bitfieldconsulting.com/posts/generics](https://bitfieldconsulting.com/posts/generics)
+
+### The smallest thing in Go `#golang` ...
+
+The smallest thing in Go `#golang`
+
+[bitfieldconsulting.com/posts/iota](https://bitfieldconsulting.com/posts/iota)
+
+### Fun with defer in `#golang`, I did't know, that ...
+
+Fun with defer in `#golang`, I did't know, that a defer object can either be heap or stack allocated. And there are some rules for inlining, too.
+
+[victoriametrics.com/blog/defer-in-go/](https://victoriametrics.com/blog/defer-in-go/)
+
+### What I like about Go is that it is still ...
+
+What I like about Go is that it is still possible to understand what's going on under the hood, whereas in JVM-based languages (for example) or dynamic languages, there are too many optimizations and abstractions. However, you don't need to know too much about how it works under the hood in Go (like memory management in C). It's just the fact that you can—you have a choice. `#golang`
+
+[blog.devtrovert.com/p/goroutine-scheduler-revealed-youll](https://blog.devtrovert.com/p/goroutine-scheduler-revealed-youll)
+
+## March 2025
+
+### Television has somewhat transformed how I work ...
+
+Television has somewhat transformed how I work in the shell on a day-to-day basis. It is especially useful for me in navigating all the local Git repositories on my laptop. I have bound Ctrl+G in my shell for that now. `#television` `#tv` `#tool` `#shell`
+
+[github.com/alexpasmantier/television](https://github.com/alexpasmantier/television)
+
+### Once in a while, I like to read a book about a ...
+
+Once in a while, I like to read a book about a programming language I have been using for a while to find new tricks or to refresh and sharpen my knowledge about it. I just finished reading "Programming Ruby 3.3," and I must say this is my favorite Ruby book now. What makes this one so special is that it is quite recent and covers all the new features. `#ruby` `#programming` `#coding`
+
+[pragprog.com/titles/ruby5/programming-ruby-3-3-5th-edition/](https://pragprog.com/titles/ruby5/programming-ruby-3-3-5th-edition/)
+
+### As you may have noticed, I like to share on ...
+
+As you may have noticed, I like to share on Mastodon and LinkedIn all the technical things I find interesting, and this blog post is technically all about that. Having said that, I love these tiny side projects. They are so relaxing to work on! `#gos` `#golang` `#tool` `#programming` `#fun`
+
+[foo.zone/gemfeed/2025-03-05-sharing-on-social-media-with-gos.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-03-05-sharing-on-social-media-with-gos.gmi)
+[foo.zone/gemfeed/2025-03-05-sharing-on-social-media-with-gos.html](https://foo.zone/gemfeed/2025-03-05-sharing-on-social-media-with-gos.html)
+
+### Personally, I think AI (LLMs) are pretty ...
+
+Personally, I think AI (LLMs) are pretty useful. But there's really some Hype around that. However, AI is about to stay - its not all hype
+
+[unixdigest.com/articles/i-passionately-hate-hype-especially-the-ai-hype.html](https://unixdigest.com/articles/i-passionately-hate-hype-especially-the-ai-hype.html)
+
+### Type aliases in `#golang`, soon also work with ...
+
+Type aliases in `#golang`, soon also work with generics. It's an interesting feature, useful for refactorings and simplifications
+
+[go.dev/blog/alias-names](https://go.dev/blog/alias-names)
+
+### `#Perl`, my "first love" of programming ...
+
+`#Perl`, my "first love" of programming languages. Still there, still use it here and then (but not as my primary language at the moment). And others do so as well, apparently. Which makes me happy! :-)
+
+[dev.to/fa5tworm/why-perl-remains-indis..-..e-of-modern-programming-languages-2io0](https://dev.to/fa5tworm/why-perl-remains-indispensable-in-the-age-of-modern-programming-languages-2io0)
+
+### I guess there are valid reasons for phttpdget, ...
+
+I guess there are valid reasons for phttpdget, which I also don't know about? Maybe complexity and/or licensing of other tools. `#FreeBSD`
+
+[l33t.codes/2024/12/05/Updating-FreeBSD-and-Re-Inventing-the-Wheel/](https://l33t.codes/2024/12/05/Updating-FreeBSD-and-Re-Inventing-the-Wheel/)
+
+### This is one of the reasons why I like ...
+
+This is one of the reasons why I like terminal-based applications so much—they are usually more lightweight than GUI-based ones (and also more flexible).
+
+[www.arp242.net/stupid-light.html](https://www.arp242.net/stupid-light.html)
+
+### Advanced Concurrency Patterns with `#Golang` ...
+
+Advanced Concurrency Patterns with `#Golang`
+
+[blogtitle.github.io/go-advanced-concurrency-patterns-part-1/](https://blogtitle.github.io/go-advanced-concurrency-patterns-part-1/)
+
+### `#SQLite` was designed as an `#TCL` extension. ...
+
+`#SQLite` was designed as an `#TCL` extension. There are ~trillion SQLite databases in active use. SQLite heavily relies on `#TCL`: C code generation via mksqlite3c.tcl, C code isn't edited directly by the SQLite developers, and for testing , and for doc generation). The devs use a custom editor written in Tcl/Tk called "e" to edit the source! There's a custom versioning system Fossil, a custom chat-room written in Tcl/Tk!
+
+[www.tcl-lang.org/community/tcl2017/assets/talk93/Paper.html](https://www.tcl-lang.org/community/tcl2017/assets/talk93/Paper.html)
+
+### Git provides automatic rendering of Markdown ...
+
+Git provides automatic rendering of Markdown files, including README.md, in a repository’s root directory" .... so much junk now in LLM powered search engines.... `#llm` `#ai`
+
+### These are some neat little Go tips. Linters ...
+
+These are some neat little Go tips. Linters already tell you when you silently omit a function return value, though. The slice filter without allocation trick is nice and simple. And I agree that switch statements are preferable to if-else statements. `#golang`
+
+[blog.devtrovert.com/p/go-ep5-avoid-contextbackground-make](https://blog.devtrovert.com/p/go-ep5-avoid-contextbackground-make)
+
+### This is a great introductory blog post about ...
+
+This is a great introductory blog post about the Helix modal editor. It's also been my first choice for over a year now. I am really looking forward to the Steel plugin system, though. I don't think I need a lot of plugins, but one or two would certainly be on my wish list. `#HelixEditor` `#Helix`
+
+[felix-knorr.net/posts/2025-03-16-helix-review.html](https://felix-knorr.net/posts/2025-03-16-helix-review.html)
+
+### Maps in Go under the hood `#golang` ...
+
+Maps in Go under the hood `#golang`
+
+[victoriametrics.com/blog/go-map/](https://victoriametrics.com/blog/go-map/)
+
+### I found that working on multiple side projects ...
+
+I found that working on multiple side projects concurrently is better than concentrating on just one. This seems inefficient, but if you to lose motivation, you can temporarily switch to another one with full élan. Remember to stop starting and start finishing. This doesn't mean you should be working on 10+ side projects concurrently! Select your projects and commit to finishing them before starting the next thing. For example, my current limit of concurrent side projects is around five.
+
+### I have been in incidents. Understandably, ...
+
+I have been in incidents. Understandably, everyone wants the issue to be resolved as quickly and others want to know how long TTR will be. IMHO, providing no estimates at all is no solution either. So maybe give a rough estimate but clearly communicate that the estimate is rough and that X, Y, and Z can interfere, meaning there is a chance it will take longer to resolve the incident. Just my thought. What's yours?
+
+[firehydrant.com/blog/hot-take-dont-provide-incident-resolution-estimates/](https://firehydrant.com/blog/hot-take-dont-provide-incident-resolution-estimates/)
+
+### I dont understand what it is. Certificates are ...
+
+I dont understand what it is. Certificates are so easy to monitor but still, expirations cause so many incidents. `#sre`
+
+[securityboulevard.com/2024/10/dont-let..-..time-prevent-outages-with-a-smart-clm/](https://securityboulevard.com/2024/10/dont-let-an-expired-certificate-cause-critical-downtime-prevent-outages-with-a-smart-clm/)
+
+### Don't just blindly trust LLMs. I recently ...
+
+Don't just blindly trust LLMs. I recently trusted an LLM, spent 1 hour debugging, and ultimately had to verify my assumption about `fcntl` behavior regarding inherited file descriptors in child processes manually with a C program, as the manual page wasn't clear to me. I could have done that immediately and I would have been done within 10 minutes. `#productivity` `#loss` `#llm` `#programming` `#C`
+
+## April 2025
+
+### I knew about any being equivalent to ...
+
+I knew about any being equivalent to interface{} in `#Golang`, but wasn't aware, that it was introduced to Go because of the generics.
+
+### Neat summary of new `#Perl` features per ...
+
+Neat summary of new `#Perl` features per release
+
+[sheet.shiar.nl/perl](https://sheet.shiar.nl/perl)
+
+### errors.As() checks for the error type, whereas ...
+
+errors.As() checks for the error type, whereas errors.Is() checks for the exact error value. Interesting read about Errors in `#golang` - and there is also a cat meme in the middle of the blog post! And then, it continues with pointers to pointers to error values or how about a pointer to an empty interface?
+
+[adrianlarion.com/golang-error-handling..-..-errors-unwrap-custom-errors-and-more/](https://adrianlarion.com/golang-error-handling-demystified-errors-is-errors-as-errors-unwrap-custom-errors-and-more/)
+
+### Good stuff: 10 years of functional options and ...
+
+Good stuff: 10 years of functional options and key lessons Learned along the way `#golang`
+
+[www.bytesizego.com/blog/10-years-functional-options-golang](https://www.bytesizego.com/blog/10-years-functional-options-golang)
+
+### I had some fun with `#FreeBSD`, `#Bhyve` and ...
+
+I had some fun with `#FreeBSD`, `#Bhyve` and `#Rocky` `#Linux`. Not just for fun, also for science and profit! `#homelab` `#selfhosting` `#self`-hosting
+
+[foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi)
+[foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html](https://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html)
+
+### The moment your blog receives PRs for typo ...
+
+The moment your blog receives PRs for typo corrections, you notice, that people are actually reading and care about your stuff :-) `#blog` `#personal` `#tech`
+
+### One thing not mentioned is that `#OpenRsync`'s ...
+
+One thing not mentioned is that `#OpenRsync`'s origin is the `#OpenBSD` project (at least as far as I am aware! Correct me if I am wrong :-) )! `#openbsd` `#rsync` `#macos` `#openrsync`
+
+[derflounder.wordpress.com/2025/04/06/r..-..laced-with-openrsync-on-macos-sequoia/](https://derflounder.wordpress.com/2025/04/06/rsync-replaced-with-openrsync-on-macos-sequoia/)
+
+### This is an interesting `#Elixir` pipes operator ...
+
+This is an interesting `#Elixir` pipes operator experiment in `#Ruby`. `#Python` has also been experimenting with such an operator. Raku (not mentioned in the linked article) already has the `==>` sequence operator, of course (which can also can be used backwards `<==` - who has doubted? :-) ). `#syntax` `#codegolf` `#fun` `#coding` `#RakuLang`
+
+[zverok.space/blog/2024-11-16-elixir-pipes.html](https://zverok.space/blog/2024-11-16-elixir-pipes.html)
+
+### The story of how my favorite `#Golang` book was ...
+
+The story of how my favorite `#Golang` book was written:
+
+[www.thecoder.cafe/p/100-go-mistakes](https://www.thecoder.cafe/p/100-go-mistakes)
+
+### These are my personal book notes from Daniel ...
+
+These are my personal book notes from Daniel Pink's "When: The Scientific Secrets of Perfect Timing." The notes are for me (to improve happiness and productivity). You still need to read the whole book to get your own insights, but maybe the notes will be useful for you as well. `#blog` `#book` `#booknotes` `#productivity`
+
+[foo.zone/gemfeed/2025-04-19-when-book-notes.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-04-19-when-book-notes.gmi)
+[foo.zone/gemfeed/2025-04-19-when-book-notes.html](https://foo.zone/gemfeed/2025-04-19-when-book-notes.html)
+
+### I certainly learned a lot reading this `#llm` ...
+
+I certainly learned a lot reading this `#llm` `#coding` `#programming`
+
+[simonwillison.net/2025/Mar/11/using-llms-for-code/](https://simonwillison.net/2025/Mar/11/using-llms-for-code/)
+
+### Writing indempotent `#Bash` scripts ...
+
+Writing indempotent `#Bash` scripts
+
+[arslan.io/2019/07/03/how-to-write-idempotent-bash-scripts/](https://arslan.io/2019/07/03/how-to-write-idempotent-bash-scripts/)
+
+### Regarding `#AI` for code generation. You should ...
+
+Regarding `#AI` for code generation. You should be at least a bit curious and exleriement a bit. You don't have to use it if you don't see fit purpose.
+
+[registerspill.thorstenball.com/p/they-..-..email=true&r=2n9ive&triedRedirect=true](https://registerspill.thorstenball.com/p/they-all-use-it?publication_id=1543843&post_id=151910861&isFreemail=true&r=2n9ive&triedRedirect=true)
+
+### I like the Rocky metaphor. And this post also ...
+
+I like the Rocky metaphor. And this post also reflects my thoughts on coding. `#llm` `#ai` `#software`
+
+[cekrem.github.io/posts/coding-as-craft-going-back-to-the-old-gym/](https://cekrem.github.io/posts/coding-as-craft-going-back-to-the-old-gym/)
+
+## May 2025
+
+### There's now also a `#Fish` shell edition of my ...
+
+There's now also a `#Fish` shell edition of my `#tmux` helper scripts: `#fishshell`
+
+[foo.zone/gemfeed/2025-05-02-terminal-multiplexing-with-tmux-fish-edition.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-05-02-terminal-multiplexing-with-tmux-fish-edition.gmi)
+[foo.zone/gemfeed/2025-05-02-terminal-multiplexing-with-tmux-fish-edition.html](https://foo.zone/gemfeed/2025-05-02-terminal-multiplexing-with-tmux-fish-edition.html)
+
+### I loved this talk. It's about how you can ...
+
+I loved this talk. It's about how you can create your own `#Linux` `#container` without Docker, using less than 100 lines of shell code without Docker or Podman and co. - Why is this talk useful? If you understand how `#containers` work "under the hood," it becomes easier to make design decisions, write your own tools, or debug production systems. I also recommend his training courses, of which I visited one once.
+
+[www.youtube.com/watch?v=4RUiVAlJE2w](https://www.youtube.com/watch?v=4RUiVAlJE2w)
+
+### Some unexpected `#golang` stuff, ppl say, that ...
+
+Some unexpected `#golang` stuff, ppl say, that Go is a simple language. IMHO the devil is in the details.
+
+[unexpected-go.com/](https://unexpected-go.com/)
+
+### With the advent of AI and LLMs, I have observed ...
+
+With the advent of AI and LLMs, I have observed that being able to type quickly has become even more important for engineers. Previously, fast typing wasn't as crucial when coding, as most of the time was spent thinking or navigating through the code. However, with LLMs, you find yourself typing much more frequently. That's an unexpected personal win for me, as I recently learned fast touch typing: `#llm` `#coding` `#programming`
+
+[foo.zone/gemfeed/2024-08-05-typing-127.1-words-per-minute.gmi (Gemini)](gemini://foo.zone/gemfeed/2024-08-05-typing-127.1-words-per-minute.gmi)
+[foo.zone/gemfeed/2024-08-05-typing-127.1-words-per-minute.html](https://foo.zone/gemfeed/2024-08-05-typing-127.1-words-per-minute.html)
+
+### For science, fun and profit, I set up a ...
+
+For science, fun and profit, I set up a `#WireGuard` mesh network for my `#FreeBSD`, `#OpenBSD`, `#RockyLinux` and `#Kubernetes` `#homelab`: There's also a mesh generator, which I wrote in `#Ruby`. `#k3s` `#linux` `#k8s` `#k3s`
+
+[foo.zone/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi (Gemini)](gemini://foo.zone/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi)
+[foo.zone/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.html](https://foo.zone/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.html)
+
+### Ever wondered about the hung task Linux ...
+
+Ever wondered about the hung task Linux messages on a busy server? Every case is unique, and there is no standard approach to debug them, but here it gets a bit demystified: `#linux` `#kernel`
+
+[blog.cloudflare.com/searching-for-the-cause-of-hung-tasks-in-the-linux-kernel/](https://blog.cloudflare.com/searching-for-the-cause-of-hung-tasks-in-the-linux-kernel/)
+
+### A bit of `#fun`: The FORTRAN hating gateway ― ...
+
+A bit of `#fun`: The FORTRAN hating gateway ― Andreas Zwinkau
+
+[beza1e1.tuxen.de/lore/fortran_hating_gateway.html](https://beza1e1.tuxen.de/lore/fortran_hating_gateway.html)
+
+### So, Golang was invented while engineers at ...
+
+So, Golang was invented while engineers at Google waited for C++ to compile. Here I am, waiting a long time for Java to compile...
+
+### I couldn't do without here-docs. If they did ...
+
+I couldn't do without here-docs. If they did not exist, I would need to find another field and pursue a career there. `#bash` `#sh` `#shell`
+
+[rednafi.com/misc/heredoc_headache/](https://rednafi.com/misc/heredoc_headache/)
+
+### I started using computers as a kid on MS-DOS ...
+
+I started using computers as a kid on MS-DOS and mainly used Norton Commander to navigate the file system in order to start games. Later, I became more interested in computing in general and switched to Linux, but there was no NC. However, there was GNU Midnight Commander, which I still use regularly to this day. It's absolutely worth checking out, even in the modern day. `#tools` `#opensource`
+
+[en.wikipedia.org/wiki/Midnight_Commander](https://en.wikipedia.org/wiki/Midnight_Commander)
+
+### Thats interesting, running `#Android` in ...
+
+Thats interesting, running `#Android` in `#Kubernetes`
+
+[ku.bz/Gs4-wpK5h](https://ku.bz/Gs4-wpK5h)
+
+### Before wiping the pre-installed `#Windows` 11 ...
+
+Before wiping the pre-installed `#Windows` 11 Pro on my new Beelink mini PC, I tested `#WSL2` with `#Fedora` `#Linux`. I compiled my pet project, I/O Riot NG (ior), which requires many system libraries, including `#BPF`. I’m impressed—everything works just like on native Fedora, and my tool runs and traces I/O syscalls with BPF out of the box. I might would prefer now Windows over MacOS if I had to chose between those two for work.
+
+[codeberg.org/snonux/ior](https://codeberg.org/snonux/ior)
+
+### Some might hate me saying this, but didnt ...
+
+Some might hate me saying this, but didnt `#systemd` solve the problem of a shared /tmp directory by introducing PrivateTmp?? but yes why did it have to go that way...
+
+[www.osnews.com/story/140968/tmp-should-not-exist/](https://www.osnews.com/story/140968/tmp-should-not-exist/)
+
+### Wouldn't still do that, even with 100% test ...
+
+Wouldn't still do that, even with 100% test coverage, LT and integration tests, unless theres an exception the business relies on `#sre`
+
+[medium.com/openclassrooms-product-desi..-..g/do-not-deploy-on-friday-92b1b46ebfe6](https://medium.com/openclassrooms-product-design-and-engineering/do-not-deploy-on-friday-92b1b46ebfe6)
+
+### Some neat slice tricks for Go: `#golang` ...
+
+Some neat slice tricks for Go: `#golang`
+
+[blog.devtrovert.com/p/12-slice-tricks-to-enhance-your-go](https://blog.devtrovert.com/p/12-slice-tricks-to-enhance-your-go)
+
+### I understand that Kubernetes is not for ...
+
+I understand that Kubernetes is not for everyone, but it still seems to be the new default for everything newly built. Despite the fact that Kubernetes is complex to maintain and use, there is still a lot of SRE/DevOps talent out there who have it on their CVs, which contributes significantly to the supportability of the infrastructure and the applications running on it. This way, you don't have to teach every new engineer your "own way" infrastructure. It's like a standard language of infrastructure that many people speak. However, Kubernetes should not be the default solution for everything, in my opinion. `#kubernetes` `#k8s`
+
+[www.gitpod.io/blog/we-are-leaving-kubernetes](https://www.gitpod.io/blog/we-are-leaving-kubernetes)
+
+## June 2025
+
+### Some great advices, will try out some of them! ...
+
+Some great advices, will try out some of them! `#programming`
+
+[endler.dev/2025/best-programmers/](https://endler.dev/2025/best-programmers/)
+
+### In `#Golang`, values are actually copied when ...
+
+In `#Golang`, values are actually copied when assigned (boxed) into an interface. That can have performance impact.
+
+[goperf.dev/01-common-patterns/interface-boxing/](https://goperf.dev/01-common-patterns/interface-boxing/)
+
+### This is a great little tutorial for searching ...
+
+This is a great little tutorial for searching in the `#HelixEditor` `#editor` `#coding`
+
+[helix-editor-tutorials.com/tutorials/using-helix-global-search/](https://helix-editor-tutorials.com/tutorials/using-helix-global-search/)
+
+### The mov instruction of a CPU is turing ...
+
+The mov instruction of a CPU is turing complete. And theres an implementation of `#Doom` only using mov, it renders one frame per 7 hours! `#fun`
+
+[beza1e1.tuxen.de/articles/accidentally_turing_complete.html](https://beza1e1.tuxen.de/articles/accidentally_turing_complete.html)
+
+### I removed the social media profile from my ...
+
+I removed the social media profile from my GrapheneOS phone. Originally, I created a separate profile just for social media to avoid using it too often. But I noticed that I switched to it too frequently. Not having social media within reach is probably the best option. `#socialmedia` `#sm` `#distractions`
+
+### So want a "real" recent UNIX? Use AIX! `#macos` ...
+
+So want a "real" recent UNIX? Use AIX! `#macos` `#unix` `#aix`
+
+[www.osnews.com/story/141633/apples-macos-unix-certification-is-a-lie/](https://www.osnews.com/story/141633/apples-macos-unix-certification-is-a-lie/)
+
+### This episode, I think, is kind of an eye-opener ...
+
+This episode, I think, is kind of an eye-opener for me personally. I knew, that AI is there to stay, but you better should now start playing with your pet projects, otherwise your performance reviews will be awkward in a year or two from now, when you are expected to use AI for your daily work. `#ai` `#llm` `#coding` `#programming`
+
+[changelog.com/friends/96](https://changelog.com/friends/96)
+
+### My `#OpenBSD` blog setup got mentioned in the ...
+
+My `#OpenBSD` blog setup got mentioned in the BSDNow.tv Podcast (In the Feedback section) :-) `#BSD` `#podcast` `#runbsd`
+
+[www.bsdnow.tv/614](https://www.bsdnow.tv/614)
+
+### `#Golang` is the best when it comes to agentic ...
+
+`#Golang` is the best when it comes to agentic coding: `#llm`
+
+[lucumr.pocoo.org/2025/6/12/agentic-coding/](https://lucumr.pocoo.org/2025/6/12/agentic-coding/)
+
+### Where `#zsh` is better than `#bash` ...
+
+Where `#zsh` is better than `#bash`
+
+[www.arp242.net/why-zsh.html](https://www.arp242.net/why-zsh.html)
+
+### I really enjoyed this talk about obscure Go ...
+
+I really enjoyed this talk about obscure Go optimizations. None of it is really standard and can change from one version of Go to another, though. `#golang` `#talk`
+
+[www.youtube.com/watch?v=rRtihWOcaLI](https://www.youtube.com/watch?v=rRtihWOcaLI)
+
+### Commenting your regular expression is generally ...
+
+Commenting your regular expression is generally a good advice! Works pretty well as described in the article not just in `#Ruby`, but also in `#Perl` (@Perl), `#RakuLang`, ...
+
+[thoughtbot.com/blog/comment-your-regular-expressions](https://thoughtbot.com/blog/comment-your-regular-expressions)
+
+### You have to make a decision for yourself, but ...
+
+You have to make a decision for yourself, but generally, work smarter (and faster—but keep the quality)! About 40 hours `#productivity` `#work` `#workload`
+
+[thesquareplanet.com/blog/about-40-hours/](https://thesquareplanet.com/blog/about-40-hours/)
+
+### "100 Go Mistakes and How to Avoid Them" is one ...
+
+"100 Go Mistakes and How to Avoid Them" is one of my favorite `#Golang` books. Julia Evans also stumbled across some issues she'd learned from this book. The book itself is an absolute must for every Gopher (or someone who wants to become one!)
+
+[jvns.ca/blog/2024/08/06/go-structs-copied-on-assignment/](https://jvns.ca/blog/2024/08/06/go-structs-copied-on-assignment/)
+
+### The `#Ruby` Data class seems quite helpful ...
+
+The `#Ruby` Data class seems quite helpful
+
+[allaboutcoding.ghinda.com/example-of-value-objects-using-rubys-data-class](https://allaboutcoding.ghinda.com/example-of-value-objects-using-rubys-data-class)
+
+Other related posts:
+
+[2025-01-01 Posts from October to December 2024](./2025-01-01-posts-from-october-to-december-2024.md)
+[2025-07-01 Posts from January to June 2025 (You are currently reading this)](./2025-07-01-posts-from-january-to-june-2025.md)
diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.md b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.md
index 0aa6d893..03577472 100644
--- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.md
+++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.md
@@ -21,18 +21,146 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h
* [⇢ ⇢ ⇢ Generating encryption keys](#generating-encryption-keys)
* [⇢ ⇢ ⇢ Configuring `zdata` ZFS pool and encryption](#configuring-zdata-zfs-pool-and-encryption)
* [⇢ ⇢ ⇢ Migrating Bhyve VMs to encrypted `bhyve` ZFS volume](#migrating-bhyve-vms-to-encrypted-bhyve-zfs-volume)
-* [⇢ ⇢ CARP](#carp)
+* [⇢ ⇢ CARP (Common Address Redundancy Protocol)](#carp-common-address-redundancy-protocol)
+* [⇢ ⇢ ⇢ How CARP Works](#how-carp-works)
+* [⇢ ⇢ ⇢ Configuring CARP](#configuring-carp)
+* [⇢ ⇢ ⇢ CARP State Change Notifications](#carp-state-change-notifications)
+* [⇢ ⇢ ZFS Replication with zrepl](#zfs-replication-with-zrepl)
+* [⇢ ⇢ ⇢ Understanding Replication Requirements](#understanding-replication-requirements)
+* [⇢ ⇢ ⇢ Why zrepl instead of HAST?](#why-zrepl-instead-of-hast)
+* [⇢ ⇢ ⇢ Installing zrepl](#installing-zrepl)
+* [⇢ ⇢ ⇢ Checking ZFS pools](#checking-zfs-pools)
+* [⇢ ⇢ ⇢ Configuring zrepl with WireGuard tunnel](#configuring-zrepl-with-wireguard-tunnel)
+* [⇢ ⇢ ⇢ Configuring zrepl on f0 (source)](#configuring-zrepl-on-f0-source)
+* [⇢ ⇢ ⇢ Configuring zrepl on f1 (sink)](#configuring-zrepl-on-f1-sink)
+* [⇢ ⇢ ⇢ Enabling and starting zrepl services](#enabling-and-starting-zrepl-services)
+* [⇢ ⇢ ⇢ Verifying replication](#verifying-replication)
+* [⇢ ⇢ ⇢ Monitoring replication](#monitoring-replication)
+* [⇢ ⇢ ⇢ A note about the Bhyve VM replication](#a-note-about-the-bhyve-vm-replication)
+* [⇢ ⇢ ⇢ Quick status check commands](#quick-status-check-commands)
+* [⇢ ⇢ ⇢ Verifying replication after reboot](#verifying-replication-after-reboot)
+* [⇢ ⇢ ⇢ Understanding Failover Limitations and Design Decisions](#understanding-failover-limitations-and-design-decisions)
+* [⇢ ⇢ ⇢# Why Manual Failover?](#-why-manual-failover)
+* [⇢ ⇢ ⇢# Current Failover Process](#-current-failover-process)
+* [⇢ ⇢ ⇢ Mounting the NFS datasets](#mounting-the-nfs-datasets)
+* [⇢ ⇢ ⇢ Failback scenario: Syncing changes from f1 back to f0](#failback-scenario-syncing-changes-from-f1-back-to-f0)
+* [⇢ ⇢ ⇢ Testing the failback scenario](#testing-the-failback-scenario)
+* [⇢ ⇢ ⇢ Troubleshooting: Files not appearing in replication](#troubleshooting-files-not-appearing-in-replication)
+* [⇢ ⇢ ⇢ Configuring automatic key loading on boot](#configuring-automatic-key-loading-on-boot)
+* [⇢ ⇢ ⇢ Troubleshooting: Replication broken due to modified destination](#troubleshooting-replication-broken-due-to-modified-destination)
+* [⇢ ⇢ ⇢ Forcing a full resync](#forcing-a-full-resync)
+* [⇢ ⇢ Future Storage Explorations](#future-storage-explorations)
+* [⇢ ⇢ ⇢ MinIO for S3-Compatible Object Storage](#minio-for-s3-compatible-object-storage)
+* [⇢ ⇢ ⇢ MooseFS for Distributed High Availability](#moosefs-for-distributed-high-availability)
+* [⇢ ⇢ NFS Server Configuration](#nfs-server-configuration)
+* [⇢ ⇢ ⇢ Setting up NFS on f0 (Primary)](#setting-up-nfs-on-f0-primary)
+* [⇢ ⇢ ⇢ Configuring Stunnel for NFS Encryption with CARP Failover](#configuring-stunnel-for-nfs-encryption-with-carp-failover)
+* [⇢ ⇢ ⇢# Why Not Native NFS over TLS?](#-why-not-native-nfs-over-tls)
+* [⇢ ⇢ ⇢# Stunnel Architecture with CARP](#-stunnel-architecture-with-carp)
+* [⇢ ⇢ ⇢# Creating a Certificate Authority for Client Authentication](#-creating-a-certificate-authority-for-client-authentication)
+* [⇢ ⇢ ⇢# Install and Configure Stunnel on f0](#-install-and-configure-stunnel-on-f0)
+* [⇢ ⇢ ⇢ Setting up NFS on f1 (Standby)](#setting-up-nfs-on-f1-standby)
+* [⇢ ⇢ ⇢ How Stunnel Works with CARP](#how-stunnel-works-with-carp)
+* [⇢ ⇢ ⇢ CARP Control Script for Clean Failover](#carp-control-script-for-clean-failover)
+* [⇢ ⇢ ⇢ CARP Management Script](#carp-management-script)
+* [⇢ ⇢ ⇢ Automatic Failback After Reboot](#automatic-failback-after-reboot)
+* [⇢ ⇢ ⇢# Why Automatic Failback?](#-why-automatic-failback)
+* [⇢ ⇢ ⇢# The Auto-Failback Script](#-the-auto-failback-script)
+* [⇢ ⇢ ⇢# Setting Up the Marker File](#-setting-up-the-marker-file)
+* [⇢ ⇢ ⇢# Configuring Cron](#-configuring-cron)
+* [⇢ ⇢ ⇢# Managing Automatic Failback](#-managing-automatic-failback)
+* [⇢ ⇢ ⇢# How It Works](#-how-it-works)
+* [⇢ ⇢ ⇢ Verifying Stunnel and CARP Status](#verifying-stunnel-and-carp-status)
+* [⇢ ⇢ ⇢ Verifying NFS Exports](#verifying-nfs-exports)
+* [⇢ ⇢ ⇢ Client Configuration for Stunnel](#client-configuration-for-stunnel)
+* [⇢ ⇢ ⇢# Preparing Client Certificates](#-preparing-client-certificates)
+* [⇢ ⇢ ⇢# Configuring Rocky Linux Clients (r0, r1, r2)](#-configuring-rocky-linux-clients-r0-r1-r2)
+* [⇢ ⇢ ⇢ Testing NFS Mount with Stunnel](#testing-nfs-mount-with-stunnel)
+* [⇢ ⇢ ⇢ Important: Encryption Keys for Replicated Datasets](#important-encryption-keys-for-replicated-datasets)
+* [⇢ ⇢ ⇢ NFS Failover with CARP and Stunnel](#nfs-failover-with-carp-and-stunnel)
+* [⇢ ⇢ ⇢ Testing CARP Failover](#testing-carp-failover)
+* [⇢ ⇢ ⇢ Handling Stale File Handles After Failover](#handling-stale-file-handles-after-failover)
+* [⇢ ⇢ ⇢ Complete Failover Test](#complete-failover-test)
+* [⇢ ⇢ ⇢ Verifying Replication Status](#verifying-replication-status)
+* [⇢ ⇢ ⇢ Post-Reboot Verification](#post-reboot-verification)
+* [⇢ ⇢ ⇢ Integration with Kubernetes](#integration-with-kubernetes)
+* [⇢ ⇢ ⇢ Security Benefits of Stunnel with Client Certificates](#security-benefits-of-stunnel-with-client-certificates)
+* [⇢ ⇢ ⇢ Laptop/Workstation Access](#laptopworkstation-access)
+* [⇢ ⇢ ⇢# Important: NFSv4 and Stunnel on Newer Linux Clients](#-important-nfsv4-and-stunnel-on-newer-linux-clients)
+* [⇢ ⇢ Mounting NFS on Rocky Linux 9](#mounting-nfs-on-rocky-linux-9)
+* [⇢ ⇢ ⇢ Installing and Configuring NFS Clients on r0, r1, and r2](#installing-and-configuring-nfs-clients-on-r0-r1-and-r2)
+* [⇢ ⇢ ⇢ Configuring Stunnel Client on All Nodes](#configuring-stunnel-client-on-all-nodes)
+* [⇢ ⇢ ⇢ Setting Up NFS Mounts](#setting-up-nfs-mounts)
+* [⇢ ⇢ ⇢ Comprehensive NFS Mount Testing](#comprehensive-nfs-mount-testing)
+* [⇢ ⇢ ⇢# Test 1: Verify Mount Status on All Nodes](#-test-1-verify-mount-status-on-all-nodes)
+* [⇢ ⇢ ⇢# Test 2: Verify Stunnel Connectivity](#-test-2-verify-stunnel-connectivity)
+* [⇢ ⇢ ⇢# Test 3: File Creation and Visibility Test](#-test-3-file-creation-and-visibility-test)
+* [⇢ ⇢ ⇢# Test 4: Verify Files on Storage Servers](#-test-4-verify-files-on-storage-servers)
+* [⇢ ⇢ ⇢# Test 5: Performance and Concurrent Access Test](#-test-5-performance-and-concurrent-access-test)
+* [⇢ ⇢ ⇢# Test 6: Directory Operations Test](#-test-6-directory-operations-test)
+* [⇢ ⇢ ⇢# Test 7: Permission and Ownership Test](#-test-7-permission-and-ownership-test)
+* [⇢ ⇢ ⇢# Test 8: Failover Test (Optional but Recommended)](#-test-8-failover-test-optional-but-recommended)
+* [⇢ ⇢ ⇢ Troubleshooting Common Issues](#troubleshooting-common-issues)
+* [⇢ ⇢ ⇢# Mount Hangs or Times Out](#-mount-hangs-or-times-out)
+* [⇢ ⇢ ⇢# Permission Denied Errors](#-permission-denied-errors)
+* [⇢ ⇢ ⇢# Files Not Visible Across Nodes](#-files-not-visible-across-nodes)
+* [⇢ ⇢ ⇢# I/O Errors When Accessing NFS Mount](#-io-errors-when-accessing-nfs-mount)
+* [⇢ ⇢ ⇢ Comprehensive Production Test Results](#comprehensive-production-test-results)
+* [⇢ ⇢ ⇢# Test Scenario: Full System Reboot and Failover](#-test-scenario-full-system-reboot-and-failover)
+* [⇢ ⇢ ⇢# Key Findings](#-key-findings)
+* [⇢ ⇢ Performance Considerations](#performance-considerations)
+* [⇢ ⇢ ⇢ Encryption Overhead](#encryption-overhead)
+* [⇢ ⇢ ⇢ Replication Bandwidth](#replication-bandwidth)
+* [⇢ ⇢ ⇢ NFS Tuning](#nfs-tuning)
+* [⇢ ⇢ ⇢ ZFS Tuning](#zfs-tuning)
+* [⇢ ⇢ ⇢ Monitoring](#monitoring)
+* [⇢ ⇢ ⇢ Cleanup After Testing](#cleanup-after-testing)
+* [⇢ ⇢ Conclusion](#conclusion)
+* [⇢ ⇢ ⇢ What We Achieved](#what-we-achieved)
+* [⇢ ⇢ ⇢ Architecture Benefits](#architecture-benefits)
+* [⇢ ⇢ ⇢ Lessons Learned](#lessons-learned)
+* [⇢ ⇢ ⇢ Next Steps](#next-steps)
+* [⇢ ⇢ ⇢ References](#references)
## Introduction
-In this blog post, we are going to extend the Beelinks with some additional storage.
+In the previous posts, we set up a FreeBSD-based Kubernetes cluster using k3s. While the base system works well, Kubernetes workloads often require persistent storage for databases, configuration files, and application data. Local storage on each node has significant limitations:
-Some photos here, describe why there are 2 different models of SSD drives (replication etc)
+* No data sharing: Pods on different nodes can't access the same data
+* Pod mobility: If a pod moves to another node, it loses access to its data
+* No redundancy: Hardware failure means data loss
+* Limited capacity: Individual nodes have finite storage
+
+This post implements a robust storage solution using:
+
+* ZFS: For data integrity, encryption, and efficient snapshots
+* CARP: For high availability with automatic IP failover
+* NFS over stunnel: For secure, encrypted network storage
+ zrepl: For continuous replication between nodes
+
+The end result is a highly available, encrypted storage system that survives node failures while providing shared storage to all Kubernetes pods. We're using two different SSD models (Samsung 870 EVO and Crucial BX500) to avoid simultaneous failures from the same manufacturing batch.
## ZFS encryption keys
+ZFS native encryption requires encryption keys to unlock datasets. We need a secure method to store these keys that balances security with operational needs:
+
+* Security: Keys must not be stored on the same disks they encrypt
+* Availability: Keys must be available at boot for automatic mounting
+* Portability: Keys should be easily moved between systems for recovery
+
+Using USB flash drives as hardware key storage provides an elegant solution. The encrypted data is unreadable without physical access to the USB key, protecting against disk theft or improper disposal. In production environments, you might use enterprise key management systems, but for a home lab, USB keys offer good security with minimal complexity.
+
### UFS on USB keys
+We'll format the USB drives with UFS (Unix File System) rather than ZFS for several reasons:
+
+* Simplicity: UFS has less overhead for small, removable media
+* Reliability: No ZFS pool import/export issues with removable devices
+
+Let's see the USB keys:
+
+TODO: Insert photos here
+
```
paul@f0:/ % doas camcontrol devlist
<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0)
@@ -49,6 +177,8 @@ paul@f1:/ % doas camcontrol devlist
paul@f1:/ %
```
+Let's create the UFS file system and mount it (done on all 3 nodes `f0`, `f1` and `f2`):
+
```sh
paul@f0:/ % doas newfs /dev/da0
/dev/da0: 15000.0MB (30720000 sectors) block size 32768, fragment size 4096
@@ -69,6 +199,9 @@ paul@f0:/ % df | grep keys
### Generating encryption keys
+The following keys will later be used to encrypt the ZFS file systems:
+
+```
paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32
paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:bhyve.key 32
paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:bhyve.key 32
@@ -80,14 +213,15 @@ paul@f0:/keys % doas chmod 400 *
paul@f0:/keys % ls -l
total 20
--r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key
--r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key
--r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key
--r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key
--r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key
--r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key
+*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key
+*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key
+*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key
+*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key
+*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key
+*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key
+````
-Copy those to all 3 nodes to /keys
+After creation, those are copied to the other two nodes `f1` and `f2` to the `/keys` partition.
### Configuring `zdata` ZFS pool and encryption
@@ -163,16 +297,48 @@ zroot/bhyve/rocky encryptionroot zroot/bhyve -
zroot/bhyve/rocky keystatus available -
```
-## CARP
+## CARP (Common Address Redundancy Protocol)
+
+High availability is crucial for storage systems. If the NFS server goes down, all pods lose access to their persistent data. CARP provides a solution by creating a virtual IP address that automatically moves between servers during failures.
+
+### How CARP Works
+
+CARP allows multiple hosts to share a virtual IP address (VIP). The hosts communicate using multicast to elect a MASTER, while others remain as BACKUP. When the MASTER fails, a BACKUP automatically promotes itself, and the VIP moves to the new MASTER. This happens within seconds, minimizing downtime.
+
+Key benefits for our storage system:
+* Automatic failover: No manual intervention required for basic failures
+* Transparent to clients: Pods continue using the same IP address
+* Works with stunnel: The VIP ensures encrypted connections follow the active server
+* Simple configuration: Just a single line in rc.conf
-adding to /etc/rc.conf on f0 and f1:
+### Configuring CARP
+
+First, add the CARP configuration to `/etc/rc.conf` on both f0 and f1:
+
+```sh
+# The virtual IP 192.168.1.138 will float between f0 and f1
ifconfig_re0_alias0="inet vhid 1 pass testpass alias 192.168.1.138/32"
+```
+
+Parameters explained:
+* `vhid 1`: Virtual Host ID - must match on all CARP members
+* `pass testpass`: Password for CARP authentication (use a stronger password in production)
+* `alias 192.168.1.138/32`: The virtual IP address with a /32 netmask
-adding to /etc/hosts:
+Next, update `/etc/hosts` on all nodes (n0, n1, n2, r0, r1, r2) to resolve the VIP hostname:
+```
192.168.1.138 f3s-storage-ha f3s-storage-ha.lan f3s-storage-ha.lan.buetow.org
+192.168.2.138 f3s-storage-ha f3s-storage-ha.wg0 f3s-storage-ha.wg0.wan.buetow.org
+```
+
+This allows clients to connect to `f3s-storage-ha` regardless of which physical server is currently the MASTER.
+
+### CARP State Change Notifications
-Adding on f0 and f1:
+To properly manage services during failover, we need to detect CARP state changes. FreeBSD's devd system can notify us when CARP transitions between MASTER and BACKUP states.
+
+Add this to `/etc/devd.conf` on both f0 and f1:
paul@f0:~ % cat <<END | doas tee -a /etc/devd.conf
notify 0 {
@@ -183,16 +349,2345 @@ notify 0 {
};
END
-next, copied that script /usr/local/bin/carpcontrol.sh and adjusted the disk to storage
+Next, create the CARP control script that will restart stunnel when CARP state changes:
+
+```sh
+paul@f0:~ % doas tee /usr/local/bin/carpcontrol.sh <<'EOF'
+#!/bin/sh
+# CARP state change handler for storage failover
+
+subsystem=$1
+state=$2
+
+logger "CARP state change: $subsystem is now $state"
+
+case "$state" in
+ MASTER)
+ # Restart stunnel to bind to the VIP
+ service stunnel restart
+ logger "Restarted stunnel for MASTER state"
+ ;;
+ BACKUP)
+ # Stop stunnel since we can't bind to VIP as BACKUP
+ service stunnel stop
+ logger "Stopped stunnel for BACKUP state"
+ ;;
+esac
+EOF
+
+paul@f0:~ % doas chmod +x /usr/local/bin/carpcontrol.sh
+
+# Copy the same script to f1
+paul@f0:~ % scp /usr/local/bin/carpcontrol.sh f1:/tmp/
+paul@f1:~ % doas mv /tmp/carpcontrol.sh /usr/local/bin/
+paul@f1:~ % doas chmod +x /usr/local/bin/carpcontrol.sh
+```
+
+Enable CARP in /boot/loader.conf:
+
+```sh
+paul@f0:~ % echo 'carp_load="YES"' | doas tee -a /boot/loader.conf
+carp_load="YES"
+paul@f1:~ % echo 'carp_load="YES"' | doas tee -a /boot/loader.conf
+carp_load="YES"
+```
+
+Then reboot both hosts or run `doas kldload carp` to load the module immediately.
+
+
+## ZFS Replication with zrepl
+
+Data replication is the cornerstone of high availability. While CARP handles IP failover, we need continuous data replication to ensure the backup server has current data when it becomes active. Without replication, failover would result in data loss or require shared storage (like iSCSI), which introduces a single point of failure.
+
+### Understanding Replication Requirements
+
+Our storage system has different replication needs:
+
+* NFS data (`/data/nfs/k3svolumes`): Contains active Kubernetes persistent volumes. Needs frequent replication (every minute) to minimize data loss during failover.
+* VM data (`/zroot/bhyve/fedora`): Contains VM images that change less frequently. Can tolerate longer replication intervals (every 10 minutes).
+
+The replication frequency determines your Recovery Point Objective (RPO) - the maximum acceptable data loss. With 1-minute replication, you lose at most 1 minute of changes during an unplanned failover.
+
+### Why zrepl instead of HAST?
+
+While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage, I've chosen zrepl for several important reasons:
+
+1. HAST can cause ZFS corruption: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.
+
+2. ZFS-aware replication: zrepl understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication.
+
+3. Snapshot history: With zrepl, you get multiple recovery points (every minute for NFS data in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.
+
+4. Easier recovery: When something goes wrong with zrepl, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary too.
+
+5. Network flexibility: zrepl works over any TCP connection (in our case, WireGuard), while HAST requires dedicated network configuration.
+
+The 5-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database - it's storage for personal projects, development work, and home lab experiments. Losing at most 5 minutes of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication.
+
+### Installing zrepl
+
+First, install zrepl on both hosts:
+
+```
+# On f0
+paul@f0:~ % doas pkg install -y zrepl
+
+# On f1
+paul@f1:~ % doas pkg install -y zrepl
+```
+
+### Checking ZFS pools
+
+Verify the pools and datasets on both hosts:
+
+```sh
+# On f0
+paul@f0:~ % doas zpool list
+NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
+zdata 928G 1.03M 928G - - 0% 0% 1.00x ONLINE -
+zroot 472G 26.7G 445G - - 0% 5% 1.00x ONLINE -
+
+paul@f0:~ % doas zfs list -r zdata/enc
+NAME USED AVAIL REFER MOUNTPOINT
+zdata/enc 200K 899G 200K /data/enc
+
+# On f1
+paul@f1:~ % doas zpool list
+NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
+zdata 928G 956K 928G - - 0% 0% 1.00x ONLINE -
+zroot 472G 11.7G 460G - - 0% 2% 1.00x ONLINE -
+
+paul@f1:~ % doas zfs list -r zdata/enc
+NAME USED AVAIL REFER MOUNTPOINT
+zdata/enc 200K 899G 200K /data/enc
+```
+
+### Configuring zrepl with WireGuard tunnel
+
+Since we have a WireGuard tunnel between f0 and f1, we'll use TCP transport over the secure tunnel instead of SSH. First, check the WireGuard IP addresses:
+
+```sh
+# Check WireGuard interface IPs
+paul@f0:~ % ifconfig wg0 | grep inet
+ inet 192.168.2.130 netmask 0xffffff00
+
+paul@f1:~ % ifconfig wg0 | grep inet
+ inet 192.168.2.131 netmask 0xffffff00
+```
+
+### Configuring zrepl on f0 (source)
+
+First, create a dedicated dataset for NFS data that will be replicated:
+
+```sh
+# Create the nfsdata dataset that will hold all data exposed via NFS
+paul@f0:~ % doas zfs create zdata/enc/nfsdata
+```
+
+Create the zrepl configuration on f0:
+
+```sh
+paul@f0:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<'EOF'
+global:
+ logging:
+ - type: stdout
+ level: info
+ format: human
+
+jobs:
+ - name: f0_to_f1_nfsdata
+ type: push
+ connect:
+ type: tcp
+ address: "192.168.2.131:8888"
+ filesystems:
+ "zdata/enc/nfsdata": true
+ send:
+ encrypted: false
+ snapshotting:
+ type: periodic
+ prefix: zrepl_
+ interval: 1m
+ pruning:
+ keep_sender:
+ - type: last_n
+ count: 10
+ keep_receiver:
+ - type: last_n
+ count: 10
+
+ - name: f0_to_f1_fedora
+ type: push
+ connect:
+ type: tcp
+ address: "192.168.2.131:8888"
+ filesystems:
+ "zroot/bhyve/fedora": true
+ send:
+ encrypted: false
+ snapshotting:
+ type: periodic
+ prefix: zrepl_
+ interval: 10m
+ pruning:
+ keep_sender:
+ - type: last_n
+ count: 10
+ keep_receiver:
+ - type: last_n
+ count: 10
+EOF
+```
+
+Key configuration notes:
+* We're using two separate replication jobs with different intervals:
+ - `f0_to_f1_nfsdata`: Replicates NFS data every minute for faster failover recovery
+ - `f0_to_f1_fedora`: Replicates Fedora VM every 10 minutes (less critical for NFS operations)
+* We're specifically replicating `zdata/enc/nfsdata` instead of the entire `zdata/enc` dataset. This dedicated dataset will contain all the data we later want to expose via NFS, keeping a clear separation between replicated NFS data and other local encrypted data.
+* The `send: encrypted: false` option disables ZFS native encryption for the replication stream. Since we're using a WireGuard tunnel between f0 and f1, the data is already encrypted in transit. Disabling ZFS stream encryption reduces CPU overhead and improves replication performance.
+
+### Configuring zrepl on f1 (sink)
+
+Create the zrepl configuration on f1:
+
+```sh
+# First create a dedicated sink dataset
+paul@f1:~ % doas zfs create zdata/sink
+
+paul@f1:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<'EOF'
+global:
+ logging:
+ - type: stdout
+ level: info
+ format: human
+
+jobs:
+ - name: "sink"
+ type: sink
+ serve:
+ type: tcp
+ listen: "192.168.2.131:8888"
+ clients:
+ "192.168.2.130": "f0"
+ recv:
+ placeholder:
+ encryption: inherit
+ root_fs: "zdata/sink"
+EOF
+```
+
+### Enabling and starting zrepl services
+
+Enable and start zrepl on both hosts:
+
+```sh
+# On f0
+paul@f0:~ % doas sysrc zrepl_enable=YES
+zrepl_enable: -> YES
+paul@f0:~ % doas service zrepl start
+Starting zrepl.
+
+# On f1
+paul@f1:~ % doas sysrc zrepl_enable=YES
+zrepl_enable: -> YES
+paul@f1:~ % doas service zrepl start
+Starting zrepl.
+```
+
+### Verifying replication
+
+Check the replication status:
+
+```sh
+# On f0, check zrepl status (use raw mode for non-tty)
+paul@f0:~ % doas zrepl status --mode raw | grep -A2 "Replication"
+"Replication":{"StartAt":"2025-07-01T22:31:48.712143123+03:00"...
+
+# Check if services are running
+paul@f0:~ % doas service zrepl status
+zrepl is running as pid 2649.
+
+paul@f1:~ % doas service zrepl status
+zrepl is running as pid 2574.
+
+# Check for zrepl snapshots on source
+paul@f0:~ % doas zfs list -t snapshot -r zdata/enc | grep zrepl
+zdata/enc@zrepl_20250701_193148_000 0B - 176K -
+
+# On f1, verify the replicated datasets
+paul@f1:~ % doas zfs list -r zdata | grep f0
+zdata/f0 576K 899G 200K none
+zdata/f0/zdata 376K 899G 200K none
+zdata/f0/zdata/enc 176K 899G 176K none
+
+# Check replicated snapshots on f1
+paul@f1:~ % doas zfs list -t snapshot -r zdata | grep zrepl
+zdata/f0/zdata/enc@zrepl_20250701_193148_000 0B - 176K -
+zdata/f0/zdata/enc@zrepl_20250701_194148_000 0B - 176K -
+```
+
+### Monitoring replication
+
+You can monitor the replication progress with:
+
+```sh
+# Real-time status
+paul@f0:~ % doas zrepl status --mode interactive
+
+# Check specific job details
+paul@f0:~ % doas zrepl status --job f0_to_f1
+```
+
+With this setup, both `zdata/enc/nfsdata` and `zroot/bhyve/fedora` on f0 will be automatically replicated to f1 every 5 minutes, with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep the last 10 snapshots while managing disk space efficiently.
+
+The replicated data appears on f1 under `zdata/sink/` with the source host and dataset hierarchy preserved:
+
+* `zdata/enc/nfsdata` → `zdata/sink/f0/zdata/enc/nfsdata`
+* `zroot/bhyve/fedora` → `zdata/sink/f0/zroot/bhyve/fedora`
+
+This is by design - zrepl preserves the complete path from the source to ensure there are no conflicts when replicating from multiple sources. The replication uses the WireGuard tunnel for secure, encrypted transport between nodes.
+
+### A note about the Bhyve VM replication
+
+While replicating a Bhyve VM (Fedora in this case) is slightly off-topic for the f3s series, I've included it here as it demonstrates zrepl's flexibility. This is a development VM I use occasionally to log in remotely for certain development tasks. Having it replicated ensures I have a backup copy available on f1 if needed.
+
+### Quick status check commands
+
+Here are the essential commands to monitor replication status:
+
+```sh
+# On the source node (f0) - check if replication is active
+paul@f0:~ % doas zrepl status --job f0_to_f1 | grep -E '(State|Last)'
+State: done
+LastError:
+
+# List all zrepl snapshots on source
+paul@f0:~ % doas zfs list -t snapshot | grep zrepl
+zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 200K -
+zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - 2.97G -
+
+# On the sink node (f1) - verify received datasets
+paul@f1:~ % doas zfs list -r zdata/sink
+NAME USED AVAIL REFER MOUNTPOINT
+zdata/sink 3.0G 896G 200K /data/sink
+zdata/sink/f0 3.0G 896G 200K none
+zdata/sink/f0/zdata 472K 896G 200K none
+zdata/sink/f0/zdata/enc 272K 896G 200K none
+zdata/sink/f0/zdata/enc/nfsdata 176K 896G 176K none
+zdata/sink/f0/zroot 2.9G 896G 200K none
+zdata/sink/f0/zroot/bhyve 2.9G 896G 200K none
+zdata/sink/f0/zroot/bhyve/fedora 2.9G 896G 2.97G none
+
+# Check received snapshots on sink
+paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep zrepl | wc -l
+ 3
+
+# Monitor replication progress in real-time (on source)
+paul@f0:~ % doas zrepl status --mode interactive
+
+# Check last replication time (on source)
+paul@f0:~ % doas zrepl status --job f0_to_f1 | grep -A1 "Replication"
+Replication:
+ Status: Idle (last run: 2025-07-01T22:41:48)
+
+# View zrepl logs for troubleshooting
+paul@f0:~ % doas tail -20 /var/log/zrepl.log | grep -E '(error|warn|replication)'
+```
+
+These commands provide a quick way to verify that:
+
+* Replication jobs are running without errors
+* Snapshots are being created on the source
+* Data is being received on the sink
+* The replication schedule is being followed
+
+### Verifying replication after reboot
+
+The zrepl service is configured to start automatically at boot. After rebooting both hosts:
+
+```sh
+paul@f0:~ % uptime
+11:17PM up 1 min, 0 users, load averages: 0.16, 0.06, 0.02
+
+paul@f0:~ % doas service zrepl status
+zrepl is running as pid 2366.
+
+paul@f1:~ % doas service zrepl status
+zrepl is running as pid 2309.
+
+# Check that new snapshots are being created and replicated
+paul@f0:~ % doas zfs list -t snapshot | grep zrepl | tail -2
+zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 200K -
+zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - 2.97G -
+
+paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep 202530
+zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 176K -
+zdata/sink/f0/zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - 2.97G -
+```
+
+The timestamps confirm that replication resumed automatically after the reboot, ensuring continuous data protection.
+
+### Understanding Failover Limitations and Design Decisions
+
+#### Why Manual Failover?
+
+This storage system intentionally uses manual failover rather than automatic failover. This might seem counterintuitive for a "high availability" system, but it's a deliberate design choice based on real-world experience:
+
+1. Split-brain prevention: Automatic failover can cause both nodes to become active simultaneously if network communication fails. This leads to data divergence that's extremely difficult to resolve.
+
+2. False positive protection: Temporary network issues or high load can trigger unwanted failovers. Manual intervention ensures failovers only occur when truly necessary.
+
+3. Data integrity over availability: For storage systems, data consistency is paramount. A few minutes of downtime is preferable to data corruption or loss.
+
+4. Simplified recovery: With manual failover, you always know which dataset is authoritative, making recovery straightforward.
+
+#### Current Failover Process
+
+The replicated datasets on f1 are intentionally not mounted (`mountpoint=none`). In case f0 fails:
+
+```sh
+# Manual steps needed on f1 to activate the replicated data:
+paul@f1:~ % doas zfs set mountpoint=/data/nfsdata zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+```
+
+However, this creates a split-brain problem: when f0 comes back online, both systems would have diverged data. Resolving this requires careful manual intervention to:
+
+1. Stop the original replication
+2. Sync changes from f1 back to f0
+3. Re-establish normal replication
+
+For true high-availability NFS, you might consider:
+
+* Shared storage (like iSCSI) with proper clustering
+* GlusterFS or similar distributed filesystems
+* Manual failover with ZFS replication (as we have here)
+
+Note: While HAST+CARP is often suggested for HA storage, it can cause filesystem corruption in practice, especially with ZFS. The block-level replication of HAST doesn't understand ZFS's transactional model, leading to inconsistent states during failover.
+
+The current zrepl setup, despite requiring manual intervention, is actually safer because:
+
+* ZFS snapshots are always consistent
+* Replication is ZFS-aware (not just block-level)
+* You have full control over the failover process
+* No risk of split-brain corruption
+
+### Mounting the NFS datasets
+
+To make the nfsdata accessible on both nodes, we need to mount them. On f0, this is straightforward:
+
+```sh
+# On f0 - set mountpoint for the primary nfsdata
+paul@f0:~ % doas zfs set mountpoint=/data/nfs zdata/enc/nfsdata
+paul@f0:~ % doas mkdir -p /data/nfs
+
+# Verify it's mounted
+paul@f0:~ % df -h /data/nfs
+Filesystem Size Used Avail Capacity Mounted on
+zdata/enc/nfsdata 899G 204K 899G 0% /data/nfs
+```
+
+On f1, we need to handle the encryption key and mount the standby copy:
+
+```sh
+# On f1 - first check encryption status
+paul@f1:~ % doas zfs get keystatus zdata/sink/f0/zdata/enc/nfsdata
+NAME PROPERTY VALUE SOURCE
+zdata/sink/f0/zdata/enc/nfsdata keystatus unavailable -
+
+# Load the encryption key (using f0's key stored on the USB)
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
+ zdata/sink/f0/zdata/enc/nfsdata
+
+# Set mountpoint and mount (same path as f0 for easier failover)
+paul@f1:~ % doas mkdir -p /data/nfs
+paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
+# Make it read-only to prevent accidental writes that would break replication
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+
+# Verify
+paul@f1:~ % df -h /data/nfs
+Filesystem Size Used Avail Capacity Mounted on
+zdata/sink/f0/zdata/enc/nfsdata 896G 204K 896G 0% /data/nfs
+```
+
+Note: The dataset is mounted at the same path (`/data/nfs`) on both hosts to simplify failover procedures. The dataset on f1 is set to `readonly=on` to prevent accidental modifications that would break replication.
+
+CRITICAL WARNING: Do NOT write to `/data/nfs/` on f1! Any modifications will break the replication. If you accidentally write to it, you'll see this error:
+
+```
+cannot receive incremental stream: destination zdata/sink/f0/zdata/enc/nfsdata has been modified
+since most recent snapshot
+```
+
+To fix a broken replication after accidental writes:
+```sh
+# Option 1: Rollback to the last common snapshot (loses local changes)
+paul@f1:~ % doas zfs rollback zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_204054_000
+
+# Option 2: Make it read-only to prevent accidents
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+```
+
+### Failback scenario: Syncing changes from f1 back to f0
+
+In a disaster recovery scenario where f0 has failed and f1 has taken over, you'll need to sync changes back when f0 returns. Here's how to failback:
+
+```sh
+# On f1: First, make the dataset writable (if it was readonly)
+paul@f1:~ % doas zfs set readonly=off zdata/sink/f0/zdata/enc/nfsdata
-/boot/loader.conf add carp_load="YES"
-reboot or run doas kldload carp0
+# Create a snapshot of the current state
+paul@f1:~ % doas zfs snapshot zdata/sink/f0/zdata/enc/nfsdata@failback
+# On f0: Stop any services using the dataset
+paul@f0:~ % doas service nfsd stop # If NFS is running
+
+# Send the snapshot from f1 to f0, forcing a rollback
+# This WILL DESTROY any data on f0 that's not on f1!
+paul@f1:~ % doas zfs send -R zdata/sink/f0/zdata/enc/nfsdata@failback | \
+ ssh f0 "doas zfs recv -F zdata/enc/nfsdata"
+
+# Alternative: If you want to see what would be received first
+paul@f1:~ % doas zfs send -R zdata/sink/f0/zdata/enc/nfsdata@failback | \
+ ssh f0 "doas zfs recv -nv -F zdata/enc/nfsdata"
+
+# After successful sync, on f0:
+paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@failback
+
+# On f1: Make it readonly again and destroy the failback snapshot
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@failback
+
+# Stop zrepl services first - CRITICAL!
+paul@f0:~ % doas service zrepl stop
+paul@f1:~ % doas service zrepl stop
+
+# Clean up any zrepl snapshots on f0
+paul@f0:~ % doas zfs list -t snapshot -r zdata/enc/nfsdata | grep zrepl | \
+ awk '{print $1}' | xargs -I {} doas zfs destroy {}
+
+# Clean up and destroy the entire replicated structure on f1
+# First release any holds
+paul@f1:~ % doas zfs holds -r zdata/sink/f0 | grep -v NAME | \
+ awk '{print $2, $1}' | while read tag snap; do
+ doas zfs release "$tag" "$snap"
+ done
+
+# Then destroy the entire f0 tree
+paul@f1:~ % doas zfs destroy -rf zdata/sink/f0
+
+# Create parent dataset structure on f1
+paul@f1:~ % doas zfs create -p zdata/sink/f0/zdata/enc
+
+# Create a fresh manual snapshot to establish baseline
+paul@f0:~ % doas zfs snapshot zdata/enc/nfsdata@manual_baseline
+
+# Send this snapshot to f1
+paul@f0:~ % doas zfs send -w zdata/enc/nfsdata@manual_baseline | \
+ ssh f1 "doas zfs recv zdata/sink/f0/zdata/enc/nfsdata"
+
+# Clean up the manual snapshot
+paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@manual_baseline
+paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@manual_baseline
+
+# Set mountpoint and make readonly on f1
+paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+
+# Load encryption key and mount on f1
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
+ zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
+# Now restart zrepl services
+paul@f0:~ % doas service zrepl start
+paul@f1:~ % doas service zrepl start
+
+# Verify replication is working
+paul@f0:~ % doas zrepl status --job f0_to_f1
+```
+
+Important notes about failback:
+
+* The `-F` flag forces a rollback on f0, destroying any local changes
+* Replication often won't resume automatically after a forced receive
+* You must clean up old zrepl snapshots on both sides
+* Creating a manual snapshot helps re-establish the replication relationship
+* Always verify replication status after the failback procedure
+* The first replication after failback will be a full send of the current state
+
+### Testing the failback scenario
+
+Here's a real test of the failback procedure:
+
+```sh
+# Simulate failure: Stop replication on f0
+paul@f0:~ % doas service zrepl stop
+
+# On f1: Take over by making the dataset writable
+paul@f1:~ % doas zfs set readonly=off zdata/sink/f0/zdata/enc/nfsdata
+
+# Write some data on f1 during the "outage"
+paul@f1:~ % echo 'Data written on f1 during failover' | doas tee /data/nfs/failover-data.txt
+Data written on f1 during failover
+
+# Now perform failback when f0 comes back online
+# Create snapshot on f1
+paul@f1:~ % doas zfs snapshot zdata/sink/f0/zdata/enc/nfsdata@failback
+
+# Send data back to f0 (note: we had to send to a temporary dataset due to holds)
+paul@f1:~ % doas zfs send -Rw zdata/sink/f0/zdata/enc/nfsdata@failback | \
+ ssh f0 "doas zfs recv -F zdata/enc/nfsdata_temp"
+
+# On f0: Rename datasets to complete failback
+paul@f0:~ % doas zfs set mountpoint=none zdata/enc/nfsdata
+paul@f0:~ % doas zfs rename zdata/enc/nfsdata zdata/enc/nfsdata_old
+paul@f0:~ % doas zfs rename zdata/enc/nfsdata_temp zdata/enc/nfsdata
+
+# Load encryption key and mount
+paul@f0:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
+paul@f0:~ % doas zfs mount zdata/enc/nfsdata
+
+# Verify the data from f1 is now on f0
+paul@f0:~ % ls -la /data/nfs/
+total 18
+drwxr-xr-x 2 root wheel 4 Jul 2 00:01 .
+drwxr-xr-x 4 root wheel 4 Jul 1 23:41 ..
+*rw-r--r-- 1 root wheel 35 Jul 2 00:01 failover-data.txt
+*rw-r--r-- 1 root wheel 12 Jul 1 23:34 hello.txt
+```
+
+Success! The failover data from f1 is now on f0. To resume normal replication, you would need to:
+
+1. Clean up old snapshots on both sides
+2. Create a new manual baseline snapshot
+3. Restart zrepl services
+
+Key learnings from the test:
+
+* The `-w` flag is essential for encrypted datasets
+* Dataset holds can complicate the process (consider sending to a temporary dataset)
+* The encryption key must be loaded after receiving the dataset
+* Always verify data integrity before resuming normal operations
+
+### Troubleshooting: Files not appearing in replication
+
+If you write files to `/data/nfs/` on f0 but they don't appear on f1, check:
+
+```sh
+# 1. Is the dataset actually mounted on f0?
+paul@f0:~ % doas zfs list -o name,mountpoint,mounted | grep nfsdata
+zdata/enc/nfsdata /data/nfs yes
+
+# If it shows "no", the dataset isn't mounted!
+# This means files are being written to the root filesystem, not ZFS
+
+# 2. Check if encryption key is loaded
+paul@f0:~ % doas zfs get keystatus zdata/enc/nfsdata
+NAME PROPERTY VALUE SOURCE
+zdata/enc/nfsdata keystatus available -
+
+# If "unavailable", load the key:
+paul@f0:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
+paul@f0:~ % doas zfs mount zdata/enc/nfsdata
+
+# 3. Verify files are in the snapshot (not just the directory)
+paul@f0:~ % ls -la /data/nfs/.zfs/snapshot/zrepl_*/
+```
+
+This issue commonly occurs after reboot if the encryption keys aren't configured to load automatically.
+
+### Configuring automatic key loading on boot
+
+To ensure all encrypted datasets are mounted automatically after reboot:
+
+```sh
+# On f0 - configure all encrypted datasets
+paul@f0:~ % doas sysrc zfskeys_enable=YES
+zfskeys_enable: NO -> YES
+paul@f0:~ % doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve"
+zfskeys_datasets: -> zdata/enc zdata/enc/nfsdata zroot/bhyve
+
+# Set correct key locations for all datasets
+paul@f0:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
+
+# On f1 - include the replicated dataset
+paul@f1:~ % doas sysrc zfskeys_enable=YES
+zfskeys_enable: NO -> YES
+paul@f1:~ % doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata"
+zfskeys_datasets: -> zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata
+
+# Set key location for replicated dataset
+paul@f1:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zdata/sink/f0/zdata/enc/nfsdata
+```
+
+Important notes:
+* Each encryption root needs its own key load entry - child datasets don't inherit key loading
+* The replicated dataset on f1 uses the same encryption key as the source on f0
+* Always verify datasets are mounted after reboot with `zfs list -o name,mounted`
+* Critical: Always ensure the replicated dataset on f1 remains read-only with `doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata`
+
+### Troubleshooting: Replication broken due to modified destination
+
+If you see the error "cannot receive incremental stream: destination has been modified since most recent snapshot", it means the read-only flag was accidentally removed on f1. To fix without a full resync:
+
+```sh
+# Stop zrepl on both servers
+paul@f0:~ % doas service zrepl stop
+paul@f1:~ % doas service zrepl stop
+
+# Find the last common snapshot
+paul@f0:~ % doas zfs list -t snapshot -o name,creation zdata/enc/nfsdata
+paul@f1:~ % doas zfs list -t snapshot -o name,creation zdata/sink/f0/zdata/enc/nfsdata
+
+# Rollback f1 to the last common snapshot (example: @zrepl_20250705_000007_000)
+paul@f1:~ % doas zfs rollback -r zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250705_000007_000
+
+# Ensure the dataset is read-only
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+
+# Restart zrepl
+paul@f0:~ % doas service zrepl start
+paul@f1:~ % doas service zrepl start
+```
+
+### Forcing a full resync
+
+If replication gets out of sync and incremental updates fail:
+
+```sh
+# Stop services
+paul@f0:~ % doas service zrepl stop
+paul@f1:~ % doas service zrepl stop
+
+# On f1: Release holds and destroy the dataset
+paul@f1:~ % doas zfs holds -r zdata/sink/f0/zdata/enc/nfsdata | \
+ grep -v NAME | awk '{print $2, $1}' | \
+ while read tag snap; do doas zfs release "$tag" "$snap"; done
+paul@f1:~ % doas zfs destroy -rf zdata/sink/f0/zdata/enc/nfsdata
+
+# On f0: Create fresh snapshot
+paul@f0:~ % doas zfs snapshot zdata/enc/nfsdata@resync
+
+# Send full dataset
+paul@f0:~ % doas zfs send -Rw zdata/enc/nfsdata@resync | \
+ ssh f1 "doas zfs recv zdata/sink/f0/zdata/enc/nfsdata"
+
+# Configure f1
+paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
+ zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
+# Clean up and restart
+paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@resync
+paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@resync
+paul@f0:~ % doas service zrepl start
+paul@f1:~ % doas service zrepl start
+```
ZFS auto scrubbing....~?
Backup of the keys on the key locations (all keys on all 3 USB keys)
+## Future Storage Explorations
+
+While zrepl provides excellent snapshot-based replication for disaster recovery, there are other storage technologies worth exploring for the f3s project:
+
+### MinIO for S3-Compatible Object Storage
+
+MinIO is a high-performance, S3-compatible object storage system that could complement our ZFS-based storage. Some potential use cases:
+
+* S3 API compatibility: Many modern applications expect S3-style object storage APIs. MinIO could provide this interface while using our ZFS storage as the backend.
+* Multi-site replication: MinIO supports active-active replication across multiple sites, which could work well with our f0/f1/f2 node setup.
+* Kubernetes native: MinIO has excellent Kubernetes integration with operators and CSI drivers, making it ideal for the f3s k3s environment.
+
+### MooseFS for Distributed High Availability
+
+MooseFS is a fault-tolerant, distributed file system that could provide true high-availability storage:
+
+* True HA: Unlike our current setup which requires manual failover, MooseFS provides automatic failover with no single point of failure.
+* POSIX compliance: Applications can use MooseFS like any regular filesystem, no code changes needed.
+* Flexible redundancy: Configure different replication levels per directory or file, optimizing storage efficiency.
+* FreeBSD support: MooseFS has native FreeBSD support, making it a natural fit for the f3s project.
+
+Both technologies could potentially run on top of our encrypted ZFS volumes, combining ZFS's data integrity and encryption features with distributed storage capabilities. This would be particularly interesting for workloads that need either S3-compatible APIs (MinIO) or transparent distributed POSIX storage (MooseFS).
+
+## NFS Server Configuration
+
+With ZFS replication in place, we can now set up NFS servers on both f0 and f1 to export the replicated data. Since native NFS over TLS (RFC 9289) has compatibility issues between Linux and FreeBSD, we'll use stunnel to provide encryption.
+
+### Setting up NFS on f0 (Primary)
+
+First, enable the NFS services in rc.conf:
+
+```sh
+paul@f0:~ % doas sysrc nfs_server_enable=YES
+nfs_server_enable: YES -> YES
+paul@f0:~ % doas sysrc nfsv4_server_enable=YES
+nfsv4_server_enable: YES -> YES
+paul@f0:~ % doas sysrc nfsuserd_enable=YES
+nfsuserd_enable: YES -> YES
+paul@f0:~ % doas sysrc mountd_enable=YES
+mountd_enable: NO -> YES
+paul@f0:~ % doas sysrc rpcbind_enable=YES
+rpcbind_enable: NO -> YES
+```
+
+Create a dedicated directory for Kubernetes volumes:
+
+```sh
+# First ensure the dataset is mounted
+paul@f0:~ % doas zfs get mounted zdata/enc/nfsdata
+NAME PROPERTY VALUE SOURCE
+zdata/enc/nfsdata mounted yes -
+
+# Create the k3svolumes directory
+paul@f0:~ % doas mkdir -p /data/nfs/k3svolumes
+paul@f0:~ % doas chmod 755 /data/nfs/k3svolumes
+
+# This directory will be replicated to f1 automatically
+```
+
+Create the /etc/exports file. Since we're using stunnel for encryption, ALL clients must connect through stunnel, which appears as localhost (127.0.0.1) to the NFS server:
+
+```sh
+paul@f0:~ % doas tee /etc/exports <<'EOF'
+V4: /data/nfs -sec=sys
+/data/nfs -alldirs -maproot=root -network 127.0.0.1 -mask 255.255.255.255
+EOF
+```
+
+The exports configuration:
+
+* `V4: /data/nfs -sec=sys`: Sets the NFSv4 root directory to /data/nfs
+* `/data/nfs -alldirs`: Allows mounting any subdirectory under /data/nfs
+* `-maproot=root`: Maps root user from client to root on server (needed for Kubernetes and ownership changes)
+* `-network 127.0.0.1`: Only accepts connections from localhost (stunnel)
+
+Note:
+* ALL clients (r0, r1, r2, laptop) must connect through stunnel for encryption
+* Stunnel proxies connections through localhost, so only 127.0.0.1 needs access
+* With NFSv4, clients mount using relative paths (e.g., `/k3svolumes` instead of `/data/nfs/k3svolumes`)
+
+Start the NFS services:
+
+```sh
+paul@f0:~ % doas service rpcbind start
+Starting rpcbind.
+paul@f0:~ % doas service mountd start
+Starting mountd.
+paul@f0:~ % doas service nfsd start
+Starting nfsd.
+paul@f0:~ % doas service nfsuserd start
+Starting nfsuserd.
+```
+
+### Configuring Stunnel for NFS Encryption with CARP Failover
+
+#### Why Not Native NFS over TLS?
+
+FreeBSD 13+ supports native NFS over TLS (RFC 9289), which would be the ideal solution. However, there are significant compatibility challenges:
+
+* Linux client support is incomplete: Most Linux distributions don't fully support NFS over TLS yet
+* Certificate management differs: FreeBSD and Linux handle TLS certificates differently for NFS
+* Kernel module requirements: Requires specific kernel modules that may not be available
+
+Stunnel provides a more compatible solution that works reliably across all operating systems while offering equivalent security.
+
+#### Stunnel Architecture with CARP
+
+Stunnel integrates seamlessly with our CARP setup:
+
+```
+ CARP VIP (192.168.1.138)
+ |
+ f0 (MASTER) ←---------→|←---------→ f1 (BACKUP)
+ stunnel:2323 | stunnel:stopped
+ nfsd:2049 | nfsd:stopped
+ |
+ Clients connect here
+```
+
+The key insight is that stunnel binds to the CARP VIP. When CARP fails over, the VIP moves to the new MASTER, and stunnel starts there automatically. Clients maintain their connection to the same IP throughout.
+
+#### Creating a Certificate Authority for Client Authentication
+
+First, create a CA to sign both server and client certificates:
+
+```sh
+# On f0 - Create CA
+paul@f0:~ % doas mkdir -p /usr/local/etc/stunnel/ca
+paul@f0:~ % cd /usr/local/etc/stunnel/ca
+paul@f0:~ % doas openssl genrsa -out ca-key.pem 4096
+paul@f0:~ % doas openssl req -new -x509 -days 3650 -key ca-key.pem -out ca-cert.pem \
+ -subj '/C=US/ST=State/L=City/O=F3S Storage/CN=F3S Stunnel CA'
+
+# Create server certificate
+paul@f0:~ % cd /usr/local/etc/stunnel
+paul@f0:~ % doas openssl genrsa -out server-key.pem 4096
+paul@f0:~ % doas openssl req -new -key server-key.pem -out server.csr \
+ -subj '/C=US/ST=State/L=City/O=F3S Storage/CN=f3s-storage-ha.lan'
+paul@f0:~ % doas openssl x509 -req -days 3650 -in server.csr -CA ca/ca-cert.pem \
+ -CAkey ca/ca-key.pem -CAcreateserial -out server-cert.pem
+
+# Create client certificates for authorized clients
+paul@f0:~ % cd /usr/local/etc/stunnel/ca
+paul@f0:~ % doas sh -c 'for client in r0 r1 r2 earth; do
+ openssl genrsa -out ${client}-key.pem 4096
+ openssl req -new -key ${client}-key.pem -out ${client}.csr \
+ -subj "/C=US/ST=State/L=City/O=F3S Storage/CN=${client}.lan.buetow.org"
+ openssl x509 -req -days 3650 -in ${client}.csr -CA ca-cert.pem \
+ -CAkey ca-key.pem -CAcreateserial -out ${client}-cert.pem
+done'
+```
+
+#### Install and Configure Stunnel on f0
+
+```sh
+# Install stunnel
+paul@f0:~ % doas pkg install -y stunnel
+
+# Configure stunnel server with client certificate authentication
+paul@f0:~ % doas tee /usr/local/etc/stunnel/stunnel.conf <<'EOF'
+cert = /usr/local/etc/stunnel/server-cert.pem
+key = /usr/local/etc/stunnel/server-key.pem
+
+setuid = stunnel
+setgid = stunnel
+
+[nfs-tls]
+accept = 192.168.1.138:2323
+connect = 127.0.0.1:2049
+CAfile = /usr/local/etc/stunnel/ca/ca-cert.pem
+verify = 2
+requireCert = yes
+EOF
+
+# Enable and start stunnel
+paul@f0:~ % doas sysrc stunnel_enable=YES
+stunnel_enable: -> YES
+paul@f0:~ % doas service stunnel start
+Starting stunnel.
+
+# Restart stunnel to apply the CARP VIP binding
+paul@f0:~ % doas service stunnel restart
+Stopping stunnel.
+Starting stunnel.
+```
+
+The configuration includes:
+* `verify = 2`: Verify client certificate and fail if not provided
+* `requireCert = yes`: Client must present a valid certificate
+* `CAfile`: Path to the CA certificate that signed the client certificates
+
+### Setting up NFS on f1 (Standby)
+
+Repeat the same configuration on f1:
+
+```sh
+paul@f1:~ % doas sysrc nfs_server_enable=YES
+nfs_server_enable: NO -> YES
+paul@f1:~ % doas sysrc nfsv4_server_enable=YES
+nfsv4_server_enable: NO -> YES
+paul@f1:~ % doas sysrc nfsuserd_enable=YES
+nfsuserd_enable: NO -> YES
+paul@f1:~ % doas sysrc mountd_enable=YES
+mountd_enable: NO -> YES
+paul@f1:~ % doas sysrc rpcbind_enable=YES
+rpcbind_enable: NO -> YES
+
+paul@f1:~ % doas tee /etc/exports <<'EOF'
+V4: /data/nfs -sec=sys
+/data/nfs -alldirs -maproot=root -network 127.0.0.1 -mask 255.255.255.255
+EOF
+
+paul@f1:~ % doas service rpcbind start
+Starting rpcbind.
+paul@f1:~ % doas service mountd start
+Starting mountd.
+paul@f1:~ % doas service nfsd start
+Starting nfsd.
+paul@f1:~ % doas service nfsuserd start
+Starting nfsuserd.
+```
+
+Configure stunnel on f1:
+
+```sh
+# Install stunnel
+paul@f1:~ % doas pkg install -y stunnel
+
+# Copy certificates from f0
+paul@f0:~ % doas tar -cf /tmp/stunnel-certs.tar -C /usr/local/etc/stunnel server-cert.pem server-key.pem ca
+paul@f0:~ % scp /tmp/stunnel-certs.tar f1:/tmp/
+paul@f1:~ % cd /usr/local/etc/stunnel && doas tar -xf /tmp/stunnel-certs.tar
+
+# Configure stunnel server on f1 with client certificate authentication
+paul@f1:~ % doas tee /usr/local/etc/stunnel/stunnel.conf <<'EOF'
+cert = /usr/local/etc/stunnel/server-cert.pem
+key = /usr/local/etc/stunnel/server-key.pem
+
+setuid = stunnel
+setgid = stunnel
+
+[nfs-tls]
+accept = 192.168.1.138:2323
+connect = 127.0.0.1:2049
+CAfile = /usr/local/etc/stunnel/ca/ca-cert.pem
+verify = 2
+requireCert = yes
+EOF
+
+# Enable and start stunnel
+paul@f1:~ % doas sysrc stunnel_enable=YES
+stunnel_enable: -> YES
+paul@f1:~ % doas service stunnel start
+Starting stunnel.
+
+# Restart stunnel to apply the CARP VIP binding
+paul@f1:~ % doas service stunnel restart
+Stopping stunnel.
+Starting stunnel.
+```
+
+### How Stunnel Works with CARP
+
+With stunnel configured to bind to the CARP VIP (192.168.1.138), only the server that is currently the CARP MASTER will accept stunnel connections. This provides automatic failover for encrypted NFS:
+
+* When f0 is CARP MASTER: stunnel on f0 accepts connections on 192.168.1.138:2323
+* When f1 becomes CARP MASTER: stunnel on f1 starts accepting connections on 192.168.1.138:2323
+* The backup server's stunnel process will fail to bind to the VIP and won't accept connections
+
+This ensures that clients always connect to the active NFS server through the CARP VIP.
+
+### CARP Control Script for Clean Failover
+
+To ensure clean failover behavior and prevent stale file handles, we'll create a control script that:
+* Stops NFS services on BACKUP nodes (preventing split-brain scenarios)
+* Starts NFS services only on the MASTER node
+* Manages stunnel binding to the CARP VIP
+
+This approach ensures clients can only connect to the active server, eliminating stale handles from the inactive server:
+
+```sh
+# Create CARP control script on both f0 and f1
+paul@f0:~ % doas tee /usr/local/bin/carpcontrol.sh <<'EOF'
+#!/bin/sh
+# CARP state change control script
+
+case "$1" in
+ MASTER)
+ logger "CARP state changed to MASTER, starting services"
+ service rpcbind start >/dev/null 2>&1
+ service mountd start >/dev/null 2>&1
+ service nfsd start >/dev/null 2>&1
+ service nfsuserd start >/dev/null 2>&1
+ service stunnel restart >/dev/null 2>&1
+ logger "CARP MASTER: NFS and stunnel services started"
+ ;;
+ BACKUP)
+ logger "CARP state changed to BACKUP, stopping services"
+ service stunnel stop >/dev/null 2>&1
+ service nfsd stop >/dev/null 2>&1
+ service mountd stop >/dev/null 2>&1
+ service nfsuserd stop >/dev/null 2>&1
+ logger "CARP BACKUP: NFS and stunnel services stopped"
+ ;;
+ *)
+ logger "CARP state changed to $1 (unhandled)"
+ ;;
+esac
+EOF
+
+paul@f0:~ % doas chmod +x /usr/local/bin/carpcontrol.sh
+
+# Add to devd configuration
+paul@f0:~ % doas tee -a /etc/devd.conf <<'EOF'
+
+# CARP state change notifications
+notify 0 {
+ match "system" "CARP";
+ match "subsystem" "[0-9]+@[a-z]+[0-9]+";
+ match "type" "(MASTER|BACKUP)";
+ action "/usr/local/bin/carpcontrol.sh $type";
+};
+EOF
+
+# Restart devd to apply changes
+paul@f0:~ % doas service devd restart
+```
+
+This enhanced script ensures that:
+* Only the MASTER node runs NFS and stunnel services
+* BACKUP nodes have all services stopped, preventing any client connections
+* Failovers are clean with no possibility of accessing the wrong server
+* Stale file handles are minimized because the old server immediately stops responding
+
+### CARP Management Script
+
+To simplify CARP state management and failover testing, create this helper script on both f0 and f1:
+
+```sh
+# Create the CARP management script
+paul@f0:~ % doas tee /usr/local/bin/carp <<'EOF'
+#!/bin/sh
+# CARP state management script
+# Usage: carp [master|backup|auto-failback enable|auto-failback disable]
+# Without arguments: shows current state
+
+# Find the interface with CARP configured
+CARP_IF=$(ifconfig -l | xargs -n1 | while read if; do
+ ifconfig "$if" 2>/dev/null | grep -q "carp:" && echo "$if" && break
+done)
+
+if [ -z "$CARP_IF" ]; then
+ echo "Error: No CARP interface found"
+ exit 1
+fi
+
+# Get CARP VHID
+VHID=$(ifconfig "$CARP_IF" | grep "carp:" | sed -n 's/.*vhid \([0-9]*\).*/\1/p')
+
+if [ -z "$VHID" ]; then
+ echo "Error: Could not determine CARP VHID"
+ exit 1
+fi
+
+# Function to get current state
+get_state() {
+ ifconfig "$CARP_IF" | grep "carp:" | awk '{print $2}'
+}
+
+# Check for auto-failback block file
+BLOCK_FILE="/data/nfs/nfs.NO_AUTO_FAILBACK"
+check_auto_failback() {
+ if [ -f "$BLOCK_FILE" ]; then
+ echo "WARNING: Auto-failback is DISABLED (file exists: $BLOCK_FILE)"
+ fi
+}
+
+# Main logic
+case "$1" in
+ "")
+ # No argument - show current state
+ STATE=$(get_state)
+ echo "CARP state on $CARP_IF (vhid $VHID): $STATE"
+ check_auto_failback
+ ;;
+ master)
+ # Force to MASTER state
+ echo "Setting CARP to MASTER state..."
+ ifconfig "$CARP_IF" vhid "$VHID" state master
+ sleep 1
+ STATE=$(get_state)
+ echo "CARP state on $CARP_IF (vhid $VHID): $STATE"
+ check_auto_failback
+ ;;
+ backup)
+ # Force to BACKUP state
+ echo "Setting CARP to BACKUP state..."
+ ifconfig "$CARP_IF" vhid "$VHID" state backup
+ sleep 1
+ STATE=$(get_state)
+ echo "CARP state on $CARP_IF (vhid $VHID): $STATE"
+ check_auto_failback
+ ;;
+ auto-failback)
+ case "$2" in
+ enable)
+ if [ -f "$BLOCK_FILE" ]; then
+ rm "$BLOCK_FILE"
+ echo "Auto-failback ENABLED (removed $BLOCK_FILE)"
+ else
+ echo "Auto-failback was already enabled"
+ fi
+ ;;
+ disable)
+ if [ ! -f "$BLOCK_FILE" ]; then
+ touch "$BLOCK_FILE"
+ echo "Auto-failback DISABLED (created $BLOCK_FILE)"
+ else
+ echo "Auto-failback was already disabled"
+ fi
+ ;;
+ *)
+ echo "Usage: $0 auto-failback [enable|disable]"
+ echo " enable: Remove block file to allow automatic failback"
+ echo " disable: Create block file to prevent automatic failback"
+ exit 1
+ ;;
+ esac
+ ;;
+ *)
+ echo "Usage: $0 [master|backup|auto-failback enable|auto-failback disable]"
+ echo " Without arguments: show current CARP state"
+ echo " master: force this node to become CARP MASTER"
+ echo " backup: force this node to become CARP BACKUP"
+ echo " auto-failback enable: allow automatic failback to f0"
+ echo " auto-failback disable: prevent automatic failback to f0"
+ exit 1
+ ;;
+esac
+EOF
+
+paul@f0:~ % doas chmod +x /usr/local/bin/carp
+
+# Copy to f1 as well
+paul@f0:~ % scp /usr/local/bin/carp f1:/tmp/
+paul@f1:~ % doas cp /tmp/carp /usr/local/bin/carp && doas chmod +x /usr/local/bin/carp
+```
+
+Now you can easily manage CARP states and auto-failback:
+
+```sh
+# Check current CARP state
+paul@f0:~ % doas carp
+CARP state on re0 (vhid 1): MASTER
+
+# If auto-failback is disabled, you'll see a warning
+paul@f0:~ % doas carp
+CARP state on re0 (vhid 1): MASTER
+WARNING: Auto-failback is DISABLED (file exists: /data/nfs/nfs.NO_AUTO_FAILBACK)
+
+# Force f0 to become BACKUP (triggers failover to f1)
+paul@f0:~ % doas carp backup
+Setting CARP to BACKUP state...
+CARP state on re0 (vhid 1): BACKUP
+
+# Disable auto-failback (useful for maintenance)
+paul@f0:~ % doas carp auto-failback disable
+Auto-failback DISABLED (created /data/nfs/nfs.NO_AUTO_FAILBACK)
+
+# Enable auto-failback
+paul@f0:~ % doas carp auto-failback enable
+Auto-failback ENABLED (removed /data/nfs/nfs.NO_AUTO_FAILBACK)
+```
+
+This enhanced script:
+- Shows warnings when auto-failback is disabled
+- Provides easy control over the auto-failback feature
+- Makes failover testing and maintenance simpler
+
+### Automatic Failback After Reboot
+
+When f0 reboots (planned or unplanned), f1 takes over as CARP MASTER. To ensure f0 automatically reclaims its primary role once it's fully operational, we'll implement an automatic failback mechanism.
+
+#### Why Automatic Failback?
+
+- **Primary node preference**: f0 has the primary storage; it should be MASTER when available
+- **Post-reboot automation**: Eliminates manual intervention after every f0 reboot
+- **Maintenance flexibility**: Can be disabled when you want f1 to remain MASTER
+
+#### The Auto-Failback Script
+
+Create this script on f0 only (not on f1):
+
+```sh
+paul@f0:~ % doas tee /usr/local/bin/carp-auto-failback.sh <<'EOF'
+#!/bin/sh
+# CARP automatic failback script for f0
+# Ensures f0 reclaims MASTER role after reboot when storage is ready
+
+LOGFILE="/var/log/carp-auto-failback.log"
+MARKER_FILE="/data/nfs/nfs.DO_NOT_REMOVE"
+BLOCK_FILE="/data/nfs/nfs.NO_AUTO_FAILBACK"
+
+log_message() {
+ echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOGFILE"
+}
+
+# Check if we're already MASTER
+CURRENT_STATE=$(/usr/local/bin/carp | awk '{print $NF}')
+if [ "$CURRENT_STATE" = "MASTER" ]; then
+ exit 0
+fi
+
+# Check if /data/nfs is mounted
+if ! mount | grep -q "on /data/nfs "; then
+ log_message "SKIP: /data/nfs not mounted"
+ exit 0
+fi
+
+# Check if marker file exists (identifies this as primary storage)
+if [ ! -f "$MARKER_FILE" ]; then
+ log_message "SKIP: Marker file $MARKER_FILE not found"
+ exit 0
+fi
+
+# Check if failback is blocked (for maintenance)
+if [ -f "$BLOCK_FILE" ]; then
+ log_message "SKIP: Failback blocked by $BLOCK_FILE"
+ exit 0
+fi
+
+# Check if NFS services are running (ensure we're fully ready)
+if ! service nfsd status >/dev/null 2>&1; then
+ log_message "SKIP: NFS services not yet running"
+ exit 0
+fi
+
+# All conditions met - promote to MASTER
+log_message "CONDITIONS MET: Promoting to MASTER (was $CURRENT_STATE)"
+/usr/local/bin/carp master
+
+# Log result
+sleep 2
+NEW_STATE=$(/usr/local/bin/carp | awk '{print $NF}')
+log_message "Failback complete: State is now $NEW_STATE"
+
+# If successful, log to system log too
+if [ "$NEW_STATE" = "MASTER" ]; then
+ logger "CARP: f0 automatically reclaimed MASTER role"
+fi
+EOF
+
+paul@f0:~ % doas chmod +x /usr/local/bin/carp-auto-failback.sh
+```
+
+#### Setting Up the Marker File
+
+The marker file identifies f0's primary storage. Create it once:
+
+```sh
+paul@f0:~ % doas touch /data/nfs/nfs.DO_NOT_REMOVE
+```
+
+This file will be replicated to f1, but since f1 mounts the dataset at a different path, it won't trigger failback there.
+
+#### Configuring Cron
+
+Add a cron job to check every minute:
+
+```sh
+paul@f0:~ % echo "* * * * * /usr/local/bin/carp-auto-failback.sh" | doas crontab -
+```
+
+#### Managing Automatic Failback
+
+The enhanced CARP script provides integrated control over auto-failback:
+
+**To temporarily disable automatic failback** (e.g., for f0 maintenance):
+```sh
+paul@f0:~ % doas carp auto-failback disable
+Auto-failback DISABLED (created /data/nfs/nfs.NO_AUTO_FAILBACK)
+```
+
+**To re-enable automatic failback**:
+```sh
+paul@f0:~ % doas carp auto-failback enable
+Auto-failback ENABLED (removed /data/nfs/nfs.NO_AUTO_FAILBACK)
+```
+
+**To check if auto-failback is enabled**:
+```sh
+paul@f0:~ % doas carp
+CARP state on re0 (vhid 1): MASTER
+# If disabled, you'll see: WARNING: Auto-failback is DISABLED
+```
+
+**To monitor failback attempts**:
+```sh
+paul@f0:~ % tail -f /var/log/carp-auto-failback.log
+```
+
+#### How It Works
+
+1. **After f0 reboots**: f1 is MASTER, f0 boots as BACKUP
+2. **Cron runs every minute**: Checks if conditions are met
+3. **Safety checks**:
+ - Is f0 currently BACKUP? (don't run if already MASTER)
+ - Is /data/nfs mounted? (ZFS datasets are ready)
+ - Does marker file exist? (confirms this is primary storage)
+ - Is failback blocked? (admin can prevent failback)
+ - Are NFS services running? (system is fully ready)
+4. **Failback occurs**: Typically 2-3 minutes after boot completes
+5. **Logging**: All attempts logged for troubleshooting
+
+This ensures f0 automatically resumes its role as primary storage server after any reboot, while providing administrative control when needed.
+
+### Verifying Stunnel and CARP Status
+
+First, check which host is currently CARP MASTER:
+
+```sh
+# On f0 - check CARP status
+paul@f0:~ % ifconfig re0 | grep carp
+ inet 192.168.1.130 netmask 0xffffff00 broadcast 192.168.1.255
+ inet 192.168.1.138 netmask 0xffffffff broadcast 192.168.1.138 vhid 1
+
+# If f0 is MASTER, verify stunnel is listening on the VIP
+paul@f0:~ % doas sockstat -l | grep 2323
+stunnel stunnel 1234 3 tcp4 192.168.1.138:2323 *:*
+
+# On f1 - check CARP status
+paul@f1:~ % ifconfig re0 | grep carp
+ inet 192.168.1.131 netmask 0xffffff00 broadcast 192.168.1.255
+
+# If f1 is BACKUP, stunnel won't be able to bind to the VIP
+paul@f1:~ % doas tail /var/log/messages | grep stunnel
+Jul 4 12:34:56 f1 stunnel: [!] bind: 192.168.1.138:2323: Can't assign requested address (49)
+```
+
+### Verifying NFS Exports
+
+Check that the exports are active on both servers:
+
+```sh
+# On f0
+paul@f0:~ % doas showmount -e localhost
+Exports list on localhost:
+/data/nfs 127.0.0.1
+
+# On f1
+paul@f1:~ % doas showmount -e localhost
+Exports list on localhost:
+/data/nfs 127.0.0.1
+```
+
+### Client Configuration for Stunnel
+
+To mount NFS shares with stunnel encryption, clients need to install and configure stunnel with their client certificates.
+
+#### Preparing Client Certificates
+
+On f0, prepare the client certificate packages:
+
+```sh
+# Create combined certificate/key files for each client
+paul@f0:~ % cd /usr/local/etc/stunnel/ca
+paul@f0:~ % doas sh -c 'for client in r0 r1 r2 earth; do
+ cat ${client}-cert.pem ${client}-key.pem > /tmp/${client}-stunnel.pem
+done'
+```
+
+#### Configuring Rocky Linux Clients (r0, r1, r2)
+
+```sh
+# Install stunnel on client (example for r0)
+[root@r0 ~]# dnf install -y stunnel
+
+# Copy client certificate and CA certificate from f0
+[root@r0 ~]# scp f0:/tmp/r0-stunnel.pem /etc/stunnel/
+[root@r0 ~]# scp f0:/usr/local/etc/stunnel/ca/ca-cert.pem /etc/stunnel/
+
+# Configure stunnel client with certificate authentication
+[root@r0 ~]# tee /etc/stunnel/stunnel.conf <<'EOF'
+cert = /etc/stunnel/r0-stunnel.pem
+CAfile = /etc/stunnel/ca-cert.pem
+client = yes
+verify = 2
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+EOF
+
+# Enable and start stunnel
+[root@r0 ~]# systemctl enable --now stunnel
+
+# Repeat for r1 and r2 with their respective certificates
+```
+
+Note: Each client must use its own certificate file (r0-stunnel.pem, r1-stunnel.pem, r2-stunnel.pem, or earth-stunnel.pem).
+
+### Testing NFS Mount with Stunnel
+
+Mount NFS through the stunnel encrypted tunnel:
+
+```sh
+# Create mount point
+[root@r0 ~]# mkdir -p /data/nfs/k3svolumes
+
+# Mount through stunnel (using localhost and NFSv4)
+[root@r0 ~]# mount -t nfs4 -o port=2323 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes
+
+# Verify mount
+[root@r0 ~]# mount | grep k3svolumes
+127.0.0.1:/data/nfs/k3svolumes on /data/nfs/k3svolumes type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=2323,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)
+
+# For persistent mount, add to /etc/fstab:
+127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0
+```
+
+Note: The mount uses localhost (127.0.0.1) because stunnel is listening locally and forwarding the encrypted traffic to the remote server.
+
+Verify the file was written and replicated:
+
+```sh
+# Check on f0
+paul@f0:~ % cat /data/nfs/test-r0.txt
+Test from r0
+
+# After replication interval (5 minutes), check on f1
+paul@f1:~ % cat /data/nfs/test-r0.txt
+Test from r0
+```
+
+### Important: Encryption Keys for Replicated Datasets
+
+When using encrypted ZFS datasets with raw sends (send -w), the replicated datasets on f1 need the encryption keys loaded to access the data:
+
+```sh
+# Check encryption status on f1
+paul@f1:~ % doas zfs get keystatus zdata/sink/f0/zdata/enc/nfsdata
+NAME PROPERTY VALUE SOURCE
+zdata/sink/f0/zdata/enc/nfsdata keystatus unavailable -
+
+# Load the encryption key (uses the same key as f0)
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/sink/f0/zdata/enc/nfsdata
+
+# Mount the dataset
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
+# Configure automatic key loading on boot
+paul@f1:~ % doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata"
+zfskeys_datasets: -> zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata
+```
+
+This ensures that after a reboot, f1 will automatically load the encryption keys and mount all encrypted datasets, including the replicated ones.
+
+### NFS Failover with CARP and Stunnel
+
+With NFS servers running on both f0 and f1 and stunnel bound to the CARP VIP:
+
+* Automatic failover: When f0 fails, CARP automatically promotes f1 to MASTER
+* Stunnel failover: The carpcontrol.sh script automatically starts stunnel on the new MASTER
+* Client transparency: Clients always connect to 192.168.1.138:2323, which routes to the active server
+* No connection disruption: Existing NFS mounts continue working through the same VIP
+* Data consistency: ZFS replication ensures f1 has recent data (within 5-minute window)
+* Read-only replica: The replicated dataset on f1 is always mounted read-only to prevent breaking replication
+* Manual intervention required for full RW failover: When f1 becomes MASTER, you must:
+ 1. Stop zrepl to prevent conflicts: `doas service zrepl stop`
+ 2. Make the replicated dataset writable: `doas zfs set readonly=off zdata/sink/f0/zdata/enc/nfsdata`
+ 3. Ensure encryption keys are loaded (should be automatic with zfskeys_enable)
+ 4. NFS will automatically start serving read/write requests through the VIP
+
+Important: The `/data/nfs` mount on f1 remains read-only during normal operation to ensure replication integrity. In case of a failover, clients can still read data immediately, but write operations require the manual steps above to promote f1 to full read-write mode.
+
+### Testing CARP Failover
+
+To test the failover process:
+
+```sh
+# On f0 (current MASTER) - trigger failover
+paul@f0:~ % doas ifconfig re0 vhid 1 state backup
+
+# On f1 - verify it becomes MASTER
+paul@f1:~ % ifconfig re0 | grep carp
+ inet 192.168.1.138 netmask 0xffffffff broadcast 192.168.1.138 vhid 1
+
+# Check stunnel is now listening on f1
+paul@f1:~ % doas sockstat -l | grep 2323
+stunnel stunnel 4567 3 tcp4 192.168.1.138:2323 *:*
+
+# On client - verify NFS mount still works
+[root@r0 ~]# ls /data/nfs/k3svolumes/
+[root@r0 ~]# echo "Test after failover" > /data/nfs/k3svolumes/failover-test.txt
+```
+
+### Handling Stale File Handles After Failover
+
+After a CARP failover, NFS clients may experience "Stale file handle" errors because they cached file handles from the previous server. To resolve this:
+
+Manual recovery (immediate fix):
+```sh
+# Force unmount and remount
+[root@r0 ~]# umount -f /data/nfs/k3svolumes
+[root@r0 ~]# mount /data/nfs/k3svolumes
+```
+
+Automatic recovery options:
+
+1. Use soft mounts with shorter timeouts in `/etc/fstab`:
+```
+127.0.0.1:/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev,soft,timeo=10,retrans=2,intr 0 0
+```
+
+2. Create an automatic recovery system using systemd timers (checks every 10 seconds):
+
+First, create the monitoring script:
+```sh
+[root@r0 ~]# cat > /usr/local/bin/check-nfs-mount.sh << 'EOF'
+#!/bin/bash
+# Fast NFS mount health monitor - runs every 10 seconds via systemd timer
+
+MOUNT_POINT="/data/nfs/k3svolumes"
+LOCK_FILE="/var/run/nfs-mount-check.lock"
+STATE_FILE="/var/run/nfs-mount.state"
+
+# Use a lock file to prevent concurrent runs
+if [ -f "$LOCK_FILE" ]; then
+ exit 0
+fi
+touch "$LOCK_FILE"
+trap "rm -f $LOCK_FILE" EXIT
+
+# Quick check - try to stat a directory with very short timeout
+if timeout 2s stat "$MOUNT_POINT" >/dev/null 2>&1; then
+ # Mount appears healthy
+ if [ -f "$STATE_FILE" ]; then
+ # Was previously unhealthy, log recovery
+ echo "NFS mount recovered at $(date)" | systemd-cat -t nfs-monitor -p info
+ rm -f "$STATE_FILE"
+ fi
+ exit 0
+fi
+
+# Mount is unhealthy
+if [ ! -f "$STATE_FILE" ]; then
+ # First detection of unhealthy state
+ echo "NFS mount unhealthy detected at $(date)" | systemd-cat -t nfs-monitor -p warning
+ touch "$STATE_FILE"
+fi
+
+# Try to fix
+echo "Attempting to fix stale NFS mount at $(date)" | systemd-cat -t nfs-monitor -p notice
+umount -f "$MOUNT_POINT" 2>/dev/null
+sleep 1
+
+if mount "$MOUNT_POINT"; then
+ echo "NFS mount fixed at $(date)" | systemd-cat -t nfs-monitor -p info
+ rm -f "$STATE_FILE"
+else
+ echo "Failed to fix NFS mount at $(date)" | systemd-cat -t nfs-monitor -p err
+fi
+EOF
+[root@r0 ~]# chmod +x /usr/local/bin/check-nfs-mount.sh
+```
+
+Create the systemd service:
+```sh
+[root@r0 ~]# cat > /etc/systemd/system/nfs-mount-monitor.service << 'EOF'
+[Unit]
+Description=NFS Mount Health Monitor
+After=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/check-nfs-mount.sh
+StandardOutput=journal
+StandardError=journal
+EOF
+```
+
+Create the systemd timer (runs every 10 seconds):
+```sh
+[root@r0 ~]# cat > /etc/systemd/system/nfs-mount-monitor.timer << 'EOF'
+[Unit]
+Description=Run NFS Mount Health Monitor every 10 seconds
+Requires=nfs-mount-monitor.service
+
+[Timer]
+OnBootSec=30s
+OnUnitActiveSec=10s
+AccuracySec=1s
+
+[Install]
+WantedBy=timers.target
+EOF
+```
+
+Enable and start the timer:
+```sh
+[root@r0 ~]# systemctl daemon-reload
+[root@r0 ~]# systemctl enable nfs-mount-monitor.timer
+[root@r0 ~]# systemctl start nfs-mount-monitor.timer
+
+# Check status
+[root@r0 ~]# systemctl status nfs-mount-monitor.timer
+● nfs-mount-monitor.timer - Run NFS Mount Health Monitor every 10 seconds
+ Loaded: loaded (/etc/systemd/system/nfs-mount-monitor.timer; enabled)
+ Active: active (waiting) since Sat 2025-07-06 10:00:00 EEST
+ Trigger: Sat 2025-07-06 10:00:10 EEST; 8s left
+
+# Monitor logs
+[root@r0 ~]# journalctl -u nfs-mount-monitor -f
+```
+
+3. For Kubernetes, use liveness probes that restart pods when NFS becomes stale
+
+Note: Stale file handles are inherent to NFS failover because file handles are server-specific. The best approach depends on your application's tolerance for brief disruptions.
+
+### Complete Failover Test
+
+Here's a comprehensive test of the failover behavior with all optimizations in place:
+
+```sh
+# 1. Check initial state
+paul@f0:~ % ifconfig re0 | grep carp
+ carp: MASTER vhid 1 advbase 1 advskew 0
+paul@f1:~ % ifconfig re0 | grep carp
+ carp: BACKUP vhid 1 advbase 1 advskew 0
+
+# 2. Create a test file from a client
+[root@r0 ~]# echo "test before failover" > /data/nfs/k3svolumes/test-before.txt
+
+# 3. Trigger failover (f0 → f1)
+paul@f0:~ % doas ifconfig re0 vhid 1 state backup
+
+# 4. Monitor client behavior
+[root@r0 ~]# ls /data/nfs/k3svolumes/
+ls: cannot access '/data/nfs/k3svolumes/': Stale file handle
+
+# 5. Check automatic recovery (within 10 seconds)
+[root@r0 ~]# journalctl -u nfs-mount-monitor -f
+Jul 06 10:15:32 r0 nfs-monitor[1234]: NFS mount unhealthy detected at Sun Jul 6 10:15:32 EEST 2025
+Jul 06 10:15:32 r0 nfs-monitor[1234]: Attempting to fix stale NFS mount at Sun Jul 6 10:15:32 EEST 2025
+Jul 06 10:15:33 r0 nfs-monitor[1234]: NFS mount fixed at Sun Jul 6 10:15:33 EEST 2025
+```
+
+Failover Timeline:
+* 0 seconds: CARP failover triggered
+* 0-2 seconds: Clients get "Stale file handle" errors (not hanging)
+* 3-10 seconds: Soft mounts ensure quick failure of operations
+* Within 10 seconds: Automatic recovery via systemd timer
+
+Benefits of the Optimized Setup:
+1. No hanging processes - Soft mounts fail quickly
+2. Clean failover - Old server stops serving immediately
+3. Fast automatic recovery - No manual intervention needed
+4. Predictable timing - Recovery within 10 seconds with systemd timer
+5. Better visibility - systemd journal provides detailed logs
+
+Important Considerations:
+* Recent writes (within 5 minutes) may not be visible after failover due to replication lag
+* Applications should handle brief NFS errors gracefully
+* For zero-downtime requirements, consider synchronous replication or distributed storage
+
+### Verifying Replication Status
+
+To check if replication is working correctly:
+
+```sh
+# Check replication status
+paul@f0:~ % doas zrepl status
+
+# Check recent snapshots on source
+paul@f0:~ % doas zfs list -t snapshot -o name,creation zdata/enc/nfsdata | tail -5
+
+# Check recent snapshots on destination
+paul@f1:~ % doas zfs list -t snapshot -o name,creation zdata/sink/f0/zdata/enc/nfsdata | tail -5
+
+# Verify data appears on f1 (should be read-only)
+paul@f1:~ % ls -la /data/nfs/k3svolumes/
+```
+
+Important: If you see "connection refused" errors in zrepl logs, ensure:
+* Both servers have zrepl running (`doas service zrepl status`)
+* No firewall or hosts.allow rules are blocking port 8888
+* WireGuard is up if using WireGuard IPs for replication
+
+### Post-Reboot Verification
+
+After rebooting the FreeBSD servers, verify the complete stack:
+
+```sh
+# Check CARP status on all servers
+paul@f0:~ % ifconfig re0 | grep carp
+paul@f1:~ % ifconfig re0 | grep carp
+
+# Verify stunnel is running on the MASTER
+paul@f0:~ % doas sockstat -l | grep 2323
+
+# Check NFS is exported
+paul@f0:~ % doas showmount -e localhost
+
+# Verify all r servers have NFS mounted
+[root@r0 ~]# mount | grep nfs
+[root@r1 ~]# mount | grep nfs
+[root@r2 ~]# mount | grep nfs
+
+# Test write access
+[root@r0 ~]# echo "Test after reboot $(date)" > /data/nfs/k3svolumes/test-reboot.txt
+
+# Verify zrepl is running and replicating
+paul@f0:~ % doas service zrepl status
+paul@f1:~ % doas service zrepl status
+```
+
+### Integration with Kubernetes
+
+In your Kubernetes manifests, you can now create PersistentVolumes using the NFS servers:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: nfs-pv
+spec:
+ capacity:
+ storage: 100Gi
+ accessModes:
+ - ReadWriteMany
+ nfs:
+ server: 192.168.1.138 # f3s-storage-ha.lan (CARP virtual IP)
+ path: /data/nfs/k3svolumes
+ mountOptions:
+ - nfsvers=4
+ - tcp
+ - hard
+ - intr
+```
+
+Using the CARP virtual IP (192.168.1.138) instead of direct server IPs ensures that Kubernetes workloads continue to access storage even if the primary NFS server fails. For encryption, configure stunnel on the Kubernetes nodes.
+
+### Security Benefits of Stunnel with Client Certificates
+
+Using stunnel with client certificate authentication for NFS encryption provides several advantages:
+
+* Compatibility: Works with any NFS version and between different operating systems
+* Strong encryption: Uses TLS/SSL with configurable cipher suites
+* Transparent: Applications don't need modification, encryption happens at transport layer
+* Performance: Minimal overhead (~2% in benchmarks)
+* Flexibility: Can encrypt any TCP-based protocol, not just NFS
+* Strong Authentication: Client certificates provide cryptographic proof of identity
+* Access Control: Only clients with valid certificates signed by your CA can connect
+* Certificate Revocation: You can revoke access by removing certificates from the CA
+
+### Laptop/Workstation Access
+
+For development workstations like "earth" (laptop), the same stunnel configuration works, but there's an important caveat with NFSv4:
+
+```sh
+# Install stunnel
+sudo dnf install stunnel
+
+# Configure stunnel (/etc/stunnel/stunnel.conf)
+cert = /etc/stunnel/earth-stunnel.pem
+CAfile = /etc/stunnel/ca-cert.pem
+client = yes
+verify = 2
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+
+# Enable and start stunnel
+sudo systemctl enable --now stunnel
+
+# Mount NFS through stunnel
+sudo mount -t nfs4 -o port=2323 127.0.0.1:/ /data/nfs
+
+# Make persistent in /etc/fstab
+127.0.0.1:/ /data/nfs nfs4 port=2323,hard,intr,_netdev 0 0
+```
+
+#### Important: NFSv4 and Stunnel on Newer Linux Clients
+
+On newer Linux distributions (like Fedora 42+), NFSv4 only uses the specified port for initial mount negotiation, but then establishes data connections directly to port 2049, bypassing stunnel. This doesn't occur on Rocky Linux 9 VMs, which properly route all traffic through the specified port.
+
+To ensure all NFS traffic goes through the encrypted tunnel on affected systems, you need to use iptables:
+
+```sh
+# Redirect all NFS traffic to the CARP VIP through stunnel
+sudo iptables -t nat -A OUTPUT -d 192.168.1.138 -p tcp --dport 2049 -j DNAT --to-destination 127.0.0.1:2323
+
+# Make it persistent (example for Fedora)
+sudo dnf install iptables-services
+sudo service iptables save
+sudo systemctl enable iptables
+
+# Or create a startup script
+cat > ~/setup-nfs-stunnel.sh << 'EOF'
+#!/bin/bash
+# Ensure NFSv4 data connections go through stunnel
+sudo iptables -t nat -D OUTPUT -d 192.168.1.138 -p tcp --dport 2049 -j DNAT --to-destination 127.0.0.1:2323 2>/dev/null
+sudo iptables -t nat -A OUTPUT -d 192.168.1.138 -p tcp --dport 2049 -j DNAT --to-destination 127.0.0.1:2323
+EOF
+chmod +x ~/setup-nfs-stunnel.sh
+```
+
+To verify all traffic is encrypted:
+```sh
+# Check active connections
+sudo ss -tnp | grep -E ":2049|:2323"
+# You should see connections to localhost:2323 (stunnel), not direct to the CARP VIP
+
+# Monitor stunnel logs
+journalctl -u stunnel -f
+# You should see connection logs for all NFS operations
+```
+
+Note: The laptop has full access to `/data/nfs` with the `-alldirs` export option, while Kubernetes nodes are restricted to `/data/nfs/k3svolumes`.
+
+The client certificate requirement ensures that:
+* Only authorized clients (r0, r1, r2, and earth) can establish stunnel connections
+* Each client has a unique identity that can be individually managed
+* Stolen IP addresses alone cannot grant access without the corresponding certificate
+* Access can be revoked without changing the server configuration
+
+The combination of ZFS encryption at rest and stunnel in transit ensures data is protected throughout its lifecycle.
+
+This configuration provides a solid foundation for shared storage in the f3s Kubernetes cluster, with automatic replication and encrypted transport.
+
+## Mounting NFS on Rocky Linux 9
+
+### Installing and Configuring NFS Clients on r0, r1, and r2
+
+First, install the necessary packages on all three Rocky Linux nodes:
+
+```sh
+# On r0, r1, and r2
+dnf install -y nfs-utils stunnel
+```
+
+### Configuring Stunnel Client on All Nodes
+
+Copy the certificate and configure stunnel on each Rocky Linux node:
+
+```sh
+# On r0
+scp f0:/usr/local/etc/stunnel/stunnel.pem /etc/stunnel/
+tee /etc/stunnel/stunnel.conf <<'EOF'
+cert = /etc/stunnel/stunnel.pem
+client = yes
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+EOF
+
+systemctl enable --now stunnel
+
+# Repeat the same configuration on r1 and r2
+```
+
+### Setting Up NFS Mounts
+
+Create mount points and configure persistent mounts on all nodes:
+
+```sh
+# On r0, r1, and r2
+mkdir -p /data/nfs/k3svolumes
+
+# Add to /etc/fstab for persistent mount (note the NFSv4 relative path)
+echo '127.0.0.1:/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,hard,intr,_netdev 0 0' >> /etc/fstab
+
+# Mount the share
+mount /data/nfs/k3svolumes
+```
+
+### Comprehensive NFS Mount Testing
+
+Here's a detailed test plan to verify NFS mounts are working correctly on all nodes:
+
+#### Test 1: Verify Mount Status on All Nodes
+
+```sh
+# On r0
+[root@r0 ~]# mount | grep k3svolumes
+# Expected output:
+# 127.0.0.1:/data/nfs/k3svolumes on /data/nfs/k3svolumes type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=2323,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)
+
+# On r1
+[root@r1 ~]# mount | grep k3svolumes
+# Should show similar output
+
+# On r2
+[root@r2 ~]# mount | grep k3svolumes
+# Should show similar output
+```
+
+#### Test 2: Verify Stunnel Connectivity
+
+```sh
+# On r0
+[root@r0 ~]# systemctl status stunnel
+# Should show: Active: active (running)
+
+[root@r0 ~]# ss -tnl | grep 2323
+# Should show: LISTEN 0 128 127.0.0.1:2323 0.0.0.0:*
+
+# Test connection to CARP VIP
+[root@r0 ~]# nc -zv 192.168.1.138 2323
+# Should show: Connection to 192.168.1.138 2323 port [tcp/*] succeeded!
+
+# Repeat on r1 and r2
+```
+
+#### Test 3: File Creation and Visibility Test
+
+```sh
+# On r0 - Create test file
+[root@r0 ~]# echo "Test from r0 - $(date)" > /data/nfs/k3svolumes/test-r0.txt
+[root@r0 ~]# ls -la /data/nfs/k3svolumes/test-r0.txt
+# Should show the file with timestamp
+
+# On r1 - Create test file and check r0's file
+[root@r1 ~]# echo "Test from r1 - $(date)" > /data/nfs/k3svolumes/test-r1.txt
+[root@r1 ~]# ls -la /data/nfs/k3svolumes/
+# Should show both test-r0.txt and test-r1.txt
+
+# On r2 - Create test file and check all files
+[root@r2 ~]# echo "Test from r2 - $(date)" > /data/nfs/k3svolumes/test-r2.txt
+[root@r2 ~]# ls -la /data/nfs/k3svolumes/
+# Should show all three files: test-r0.txt, test-r1.txt, test-r2.txt
+```
+
+#### Test 4: Verify Files on Storage Servers
+
+```sh
+# On f0 (primary storage)
+paul@f0:~ % ls -la /data/nfs/k3svolumes/
+# Should show all three test files
+
+# Wait 5 minutes for replication, then check on f1
+paul@f1:~ % ls -la /data/nfs/k3svolumes/
+# Should show all three test files (after replication)
+```
+
+#### Test 5: Performance and Concurrent Access Test
+
+```sh
+# On r0 - Write large file
+[root@r0 ~]# dd if=/dev/zero of=/data/nfs/k3svolumes/test-large-r0.dat bs=1M count=100
+# Should complete without errors
+
+# On r1 - Read the file while r2 writes
+[root@r1 ~]# dd if=/data/nfs/k3svolumes/test-large-r0.dat of=/dev/null bs=1M &
+# Simultaneously on r2
+[root@r2 ~]# dd if=/dev/zero of=/data/nfs/k3svolumes/test-large-r2.dat bs=1M count=100
+
+# Check for any errors or performance issues
+```
+
+#### Test 6: Directory Operations Test
+
+```sh
+# On r0 - Create directory structure
+[root@r0 ~]# mkdir -p /data/nfs/k3svolumes/test-dir/subdir1/subdir2
+[root@r0 ~]# echo "Deep file" > /data/nfs/k3svolumes/test-dir/subdir1/subdir2/deep.txt
+
+# On r1 - Verify and add files
+[root@r1 ~]# ls -la /data/nfs/k3svolumes/test-dir/subdir1/subdir2/
+[root@r1 ~]# echo "Another file from r1" > /data/nfs/k3svolumes/test-dir/subdir1/file-r1.txt
+
+# On r2 - Verify complete structure
+[root@r2 ~]# find /data/nfs/k3svolumes/test-dir -type f
+# Should show both files
+```
+
+#### Test 7: Permission and Ownership Test
+
+```sh
+# On r0 - Create files with different permissions
+[root@r0 ~]# touch /data/nfs/k3svolumes/test-perms-644.txt
+[root@r0 ~]# chmod 644 /data/nfs/k3svolumes/test-perms-644.txt
+[root@r0 ~]# touch /data/nfs/k3svolumes/test-perms-755.txt
+[root@r0 ~]# chmod 755 /data/nfs/k3svolumes/test-perms-755.txt
+
+# On r1 and r2 - Verify permissions are preserved
+[root@r1 ~]# ls -l /data/nfs/k3svolumes/test-perms-*.txt
+[root@r2 ~]# ls -l /data/nfs/k3svolumes/test-perms-*.txt
+# Permissions should match what was set on r0
+```
+
+#### Test 8: Failover Test (Optional but Recommended)
+
+```sh
+# On f0 - Trigger CARP failover
+paul@f0:~ % doas ifconfig re0 vhid 1 state backup
+
+# On all Rocky nodes - Verify mounts still work
+[root@r0 ~]# echo "Test during failover from r0 - $(date)" > /data/nfs/k3svolumes/failover-test-r0.txt
+[root@r1 ~]# echo "Test during failover from r1 - $(date)" > /data/nfs/k3svolumes/failover-test-r1.txt
+[root@r2 ~]# echo "Test during failover from r2 - $(date)" > /data/nfs/k3svolumes/failover-test-r2.txt
+
+# Verify all files are accessible
+[root@r0 ~]# ls -la /data/nfs/k3svolumes/failover-test-*.txt
+
+# On f1 - Verify it's now MASTER
+paul@f1:~ % ifconfig re0 | grep carp
+# Should show the VIP 192.168.1.138
+
+# Restore f0 as MASTER
+paul@f0:~ % doas ifconfig re0 vhid 1 state master
+```
+
+### Troubleshooting Common Issues
+
+#### Mount Hangs or Times Out
+
+```sh
+# Check stunnel connectivity
+systemctl status stunnel
+ss -tnl | grep 2323
+telnet 127.0.0.1 2323
+
+# Check if you can reach the CARP VIP
+ping 192.168.1.138
+nc -zv 192.168.1.138 2323
+
+# Check for firewall issues
+iptables -L -n | grep 2323
+```
+
+#### Permission Denied Errors
+
+```sh
+# Verify the export allows your IP
+# On f0 or f1
+doas showmount -e localhost
+
+# Check if SELinux is blocking (on Rocky Linux)
+getenforce
+# If enforcing, try:
+setenforce 0 # Temporary for testing
+# Or add proper SELinux context:
+setsebool -P use_nfs_home_dirs 1
+```
+
+#### Files Not Visible Across Nodes
+
+```sh
+# Force NFS cache refresh
+# On the affected node
+umount /data/nfs/k3svolumes
+mount /data/nfs/k3svolumes
+
+# Check NFS version
+nfsstat -m
+# Should show NFSv4
+```
+
+#### I/O Errors When Accessing NFS Mount
+
+I/O errors can have several causes:
+
+1. Missing localhost in exports (most common with stunnel):
+ - Since stunnel proxies connections, the NFS server sees requests from 127.0.0.1
+ - Ensure your exports include localhost access:
+ ```
+ /data/nfs/k3svolumes -maproot=root -network 127.0.0.1 -mask 255.255.255.255
+ ```
+
+2. Stunnel connection issues or CARP failover:
+
+```sh
+# On the affected node (e.g., r0)
+# Check stunnel is running
+systemctl status stunnel
+
+# Restart stunnel to re-establish connection
+systemctl restart stunnel
+
+# Force remount
+umount -f -l /data/nfs/k3svolumes
+mount -t nfs4 -o port=2323,hard,intr 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes
+
+# Check which FreeBSD host is CARP MASTER
+# On f0
+ssh f0 "ifconfig re0 | grep carp"
+# On f1
+ssh f1 "ifconfig re0 | grep carp"
+
+# Verify stunnel on MASTER is bound to VIP
+# On the MASTER host
+ssh <master-host> "sockstat -l | grep 2323"
+
+# Debug stunnel connection
+openssl s_client -connect 192.168.1.138:2323 </dev/null
+
+# If persistent I/O errors, check logs
+journalctl -u stunnel -n 50
+dmesg | tail -20 | grep -i nfs
+```
+
+### Comprehensive Production Test Results
+
+After implementing all the improvements (enhanced CARP control script, soft mounts, and automatic recovery), here's a complete test of the setup including reboots and failovers:
+
+#### Test Scenario: Full System Reboot and Failover
+
+```
+1. Initial state: Rebooted all servers (f0, f1, f2)
+ - Result: f1 became CARP MASTER after reboot (not always f0)
+ - NFS accessible and writable from all clients
+
+2. Created test file from laptop:
+ paul@earth:~ % echo "Post-reboot test at $(date)" > /data/nfs/k3svolumes/reboot-test.txt
+
+3. Verified 1-minute replication to f1:
+ - File appeared on f1 within 70 seconds
+ - Content identical on both servers
+
+4. Performed failover from f0 to f1:
+ paul@f0:~ % doas ifconfig re0 vhid 1 state backup
+ - f1 immediately became MASTER
+ - Clients experienced "Stale file handle" errors
+ - With soft mounts: No hanging, immediate error response
+
+5. Recovery time:
+ - Manual recovery: Immediate with umount/mount
+ - Automatic recovery: Within 10 seconds via systemd timer
+ - No data loss during failover
+
+6. Failback to f0:
+ paul@f1:~ % doas ifconfig re0 vhid 1 state backup
+ - f0 reclaimed MASTER status
+ - Similar stale handle behavior
+ - Recovery within 10 seconds
+```
+
+#### Key Findings
+
+1. CARP Master Selection: After reboot, either f0 or f1 can become MASTER. This is normal CARP behavior and doesn't affect functionality.
+
+2. Stale File Handles: Despite all optimizations, NFS clients still experience stale file handles during failover. This is inherent to NFS protocol design. However:
+ - Soft mounts prevent hanging
+ - Automatic recovery works reliably
+ - No data loss occurs
+
+3. Replication Timing: The 1-minute replication interval for NFS data ensures minimal data loss window during unplanned failovers. The Fedora VM replication runs every 10 minutes, which is sufficient for less critical VM data.
+
+4. Service Management: The enhanced carpcontrol.sh script successfully stops services on BACKUP nodes, preventing split-brain scenarios.
+
+## Performance Considerations
+
+### Encryption Overhead
+
+Stunnel adds CPU overhead for TLS encryption/decryption. On modern hardware, the impact is minimal:
+
+* Beelink Mini PCs: With hardware AES acceleration, expect 5-10% CPU overhead
+* Network throughput: Gigabit Ethernet is usually the bottleneck, not TLS
+* Latency: Adds <1ms in LAN environments
+
+For reference, with AES-256-GCM on a typical mini PC:
+* Sequential reads: ~110 MB/s (near line-speed for gigabit)
+* Sequential writes: ~105 MB/s
+* Random 4K IOPS: ~15% reduction compared to unencrypted
+
+### Replication Bandwidth
+
+ZFS replication with zrepl is efficient, only sending changed blocks:
+
+* Initial sync: Full dataset size (can be large)
+* Incremental: Typically <1% of dataset size per snapshot
+* Network usage: With 1-minute intervals and moderate changes, expect 10-50 MB/minute
+
+To monitor replication bandwidth:
+```sh
+# On f0, check network usage on WireGuard interface
+doas systat -ifstat 1
+# Look for wg0 traffic during replication
+```
+
+### NFS Tuning
+
+For optimal performance with Kubernetes workloads:
+
+```sh
+# On NFS server (f0/f1) - /etc/sysctl.conf
+vfs.nfsd.async=1 # Enable async writes (careful with data integrity)
+vfs.nfsd.cachetcp=1 # Cache TCP connections
+vfs.nfsd.tcphighwater=64 # Increase TCP connection limit
+
+# On NFS clients - mount options
+rsize=131072,wsize=131072 # Larger read/write buffers
+hard,intr # Hard mount with interruption
+vers=4.2 # Use latest NFSv4.2 for best performance
+```
+
+### ZFS Tuning
+
+Key ZFS settings for NFS storage:
+
+```sh
+# Set on the NFS dataset
+zfs set compression=lz4 zdata/enc/nfsdata # Fast compression
+zfs set atime=off zdata/enc/nfsdata # Disable access time updates
+zfs set redundant_metadata=most zdata/enc/nfsdata # Protect metadata
+```
+
+### Monitoring
+
+Monitor system performance to identify bottlenecks:
+
+```sh
+# CPU and memory
+doas top -P
+
+# Disk I/O
+doas gstat -p
+
+# Network traffic
+doas netstat -w 1 -h
+
+# ZFS statistics
+doas zpool iostat -v 1
+
+# NFS statistics
+doas nfsstat -s -w 1
+```
+
+### Cleanup After Testing
+
+```sh
+# Remove test files (run on any node)
+rm -f /data/nfs/k3svolumes/test-*.txt
+rm -f /data/nfs/k3svolumes/test-large-*.dat
+rm -f /data/nfs/k3svolumes/failover-test-*.txt
+rm -f /data/nfs/k3svolumes/test-perms-*.txt
+rm -rf /data/nfs/k3svolumes/test-dir
+```
+
+This comprehensive testing ensures that:
+* All nodes can mount the NFS share
+* Files created on one node are visible on all others
+* The encrypted stunnel connection is working
+* Permissions and ownership are preserved
+* The setup can handle concurrent access
+* Failover works correctly (if tested)
+
+## Conclusion
+
+We've built a robust, encrypted storage system for our FreeBSD-based Kubernetes cluster that provides:
+
+### What We Achieved
+
+* High Availability: CARP ensures the storage VIP moves automatically during failures
+* Data Protection: ZFS encryption protects data at rest, stunnel protects data in transit
+* Continuous Replication: 1-minute RPO for critical data, automated via zrepl
+* Secure Access: Client certificate authentication prevents unauthorized access
+* Kubernetes Integration: Shared storage accessible from all cluster nodes
+
+### Architecture Benefits
+
+This design prioritizes data integrity over pure availability:
+* Manual failover prevents split-brain scenarios
+* Certificate-based authentication provides strong security
+* Encrypted replication protects data even over untrusted networks
+* ZFS snapshots enable point-in-time recovery
+
+### Lessons Learned
+
+1. Stunnel vs Native NFS/TLS: While native encryption would be ideal, stunnel provides better cross-platform compatibility
+2. Manual vs Automatic Failover: For storage systems, controlled failover often prevents more problems than it causes
+3. Replication Frequency: Balance between data protection (RPO) and system load
+4. Client Compatibility: Different NFS implementations behave differently - test thoroughly
+
+### Next Steps
+
+With reliable storage in place, we can now:
+* Deploy stateful applications on Kubernetes
+* Set up databases with persistent volumes
+* Create shared configuration stores
+* Implement backup strategies using ZFS snapshots
+
+The storage layer is the foundation for any serious Kubernetes deployment. By building it on FreeBSD with ZFS, CARP, and stunnel, we get enterprise-grade features on commodity hardware.
+
+### References
+
+* FreeBSD CARP documentation: https://docs.freebsd.org/en/books/handbook/advanced-networking/#carp
+* ZFS encryption guide: https://docs.freebsd.org/en/books/handbook/zfs/#zfs-encryption
+* Stunnel documentation: https://www.stunnel.org/docs.html
+* zrepl documentation: https://zrepl.github.io/
+
Other *BSD-related posts:
[2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md)
@@ -209,10 +2704,3 @@ Other *BSD-related posts:
E-Mail your comments to `paul@nospam.buetow.org`
[Back to the main site](../)
-
-https://forums.freebsd.org/threads/hast-and-zfs-with-carp-failover.29639/
-
-
-E-Mail your comments to `paul@nospam.buetow.org`
-
-[Back to the main site](../)
diff --git a/gemfeed/configure-stunnel-nfs-r1-r2.sh b/gemfeed/configure-stunnel-nfs-r1-r2.sh
new file mode 100755
index 00000000..58431744
--- /dev/null
+++ b/gemfeed/configure-stunnel-nfs-r1-r2.sh
@@ -0,0 +1,163 @@
+#!/bin/sh
+# Configure stunnel and NFS mounts on r1 and r2 for f3s storage
+# This script should be run on both r1 and r2 Rocky Linux systems
+
+set -e
+
+# Function to print colored status messages
+print_status() {
+ echo -e "\n\033[1;34m>>> $1\033[0m"
+}
+
+print_error() {
+ echo -e "\033[1;31mERROR: $1\033[0m"
+ exit 1
+}
+
+print_success() {
+ echo -e "\033[1;32m✓ $1\033[0m"
+}
+
+# Check if running as root
+if [ "$EUID" -ne 0 ]; then
+ print_error "Please run as root"
+fi
+
+# Detect hostname
+HOSTNAME=$(hostname -s)
+if [ "$HOSTNAME" != "r1" ] && [ "$HOSTNAME" != "r2" ]; then
+ print_error "This script should only be run on r1 or r2"
+fi
+
+print_status "Configuring stunnel and NFS on $HOSTNAME"
+
+# Step 1: Install stunnel package
+print_status "Installing stunnel package..."
+dnf install -y stunnel || print_error "Failed to install stunnel"
+print_success "stunnel installed"
+
+# Step 2: Create stunnel directory
+print_status "Creating stunnel configuration directory..."
+mkdir -p /etc/stunnel
+print_success "Directory created"
+
+# Step 3: Copy stunnel certificate from f0
+print_status "Copying stunnel certificate from f0..."
+echo "Please copy the certificate manually:"
+echo "On f0, run: scp /usr/local/etc/stunnel/stunnel.pem root@$HOSTNAME:/etc/stunnel/"
+echo ""
+echo "Press Enter when the certificate has been copied..."
+read -r
+
+# Verify certificate exists
+if [ ! -f /etc/stunnel/stunnel.pem ]; then
+ print_error "Certificate not found at /etc/stunnel/stunnel.pem"
+fi
+print_success "Certificate found"
+
+# Step 4: Create stunnel client configuration
+print_status "Creating stunnel client configuration..."
+cat > /etc/stunnel/stunnel.conf <<'EOF'
+cert = /etc/stunnel/stunnel.pem
+client = yes
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+EOF
+print_success "Stunnel configuration created"
+
+# Step 5: Create systemd service for stunnel
+print_status "Creating stunnel systemd service..."
+cat > /etc/systemd/system/stunnel.service <<'EOF'
+[Unit]
+Description=SSL tunnel for network daemons
+After=network.target
+
+[Service]
+Type=forking
+ExecStart=/usr/bin/stunnel /etc/stunnel/stunnel.conf
+ExecStop=/usr/bin/killall stunnel
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+EOF
+print_success "Systemd service created"
+
+# Step 6: Enable and start stunnel service
+print_status "Enabling and starting stunnel service..."
+systemctl daemon-reload
+systemctl enable stunnel
+systemctl start stunnel
+systemctl status stunnel --no-pager || print_error "Stunnel failed to start"
+print_success "Stunnel service is running"
+
+# Step 7: Create mount point for NFS
+print_status "Creating NFS mount point..."
+mkdir -p /data/nfs/k3svolumes
+print_success "Mount point created"
+
+# Step 8: Test mount NFS through stunnel
+print_status "Testing NFS mount through stunnel..."
+mount -t nfs4 -o port=2323 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes || print_error "Failed to mount NFS"
+print_success "NFS mounted successfully"
+
+# Step 9: Verify the mount
+print_status "Verifying NFS mount..."
+mount | grep k3svolumes || print_error "NFS mount not found"
+df -h /data/nfs/k3svolumes
+print_success "NFS mount verified"
+
+# Step 10: Unmount for fstab configuration
+print_status "Unmounting NFS to configure fstab..."
+umount /data/nfs/k3svolumes
+print_success "Unmounted"
+
+# Step 11: Add entry to /etc/fstab for persistent mount
+print_status "Adding NFS mount to /etc/fstab..."
+# Check if entry already exists
+if grep -q "127.0.0.1:/data/nfs/k3svolumes" /etc/fstab; then
+ print_status "Entry already exists in /etc/fstab"
+else
+ echo "127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0" >> /etc/fstab
+ print_success "Added to /etc/fstab"
+fi
+
+# Step 12: Mount using fstab
+print_status "Mounting NFS using fstab entry..."
+mount /data/nfs/k3svolumes || print_error "Failed to mount from fstab"
+print_success "Mounted from fstab"
+
+# Step 13: Final verification
+print_status "Final verification..."
+echo ""
+echo "Stunnel status:"
+systemctl is-active stunnel || print_error "Stunnel is not active"
+echo ""
+echo "NFS mount status:"
+mount | grep k3svolumes
+echo ""
+echo "Disk usage:"
+df -h /data/nfs/k3svolumes
+echo ""
+
+# Step 14: Test write access
+print_status "Testing write access..."
+TEST_FILE="/data/nfs/k3svolumes/test-$HOSTNAME-$(date +%s).txt"
+echo "Test from $HOSTNAME at $(date)" > "$TEST_FILE" || print_error "Failed to write test file"
+cat "$TEST_FILE"
+print_success "Write test successful"
+
+print_success "Configuration complete on $HOSTNAME!"
+echo ""
+echo "Summary:"
+echo "- Stunnel is configured as a client connecting to 192.168.1.138:2323 (CARP VIP)"
+echo "- NFS is mounted through stunnel at /data/nfs/k3svolumes"
+echo "- Mount is persistent across reboots (configured in /etc/fstab)"
+echo "- All traffic between this host and the NFS server is encrypted"
+echo ""
+echo "To verify everything after reboot:"
+echo " systemctl status stunnel"
+echo " mount | grep k3svolumes"
+echo " ls -la /data/nfs/k3svolumes/" \ No newline at end of file
diff --git a/gemfeed/index.md b/gemfeed/index.md
index d8b338a2..e1fa7994 100644
--- a/gemfeed/index.md
+++ b/gemfeed/index.md
@@ -2,6 +2,7 @@
## To be in the .zone!
+[2025-07-01 - Posts from January to June 2025](./2025-07-01-posts-from-january-to-june-2025.md)
[2025-06-22 - Task Samurai: An agentic coding learning experiment](./2025-06-22-task-samurai.md)
[2025-06-07 - 'A Monk's Guide to Happiness' book notes](./2025-06-07-a-monks-guide-to-happiness-book-notes.md)
[2025-05-11 - f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md)
diff --git a/gemfeed/stunnel-nfs-quick-reference.txt b/gemfeed/stunnel-nfs-quick-reference.txt
new file mode 100644
index 00000000..ca7f577a
--- /dev/null
+++ b/gemfeed/stunnel-nfs-quick-reference.txt
@@ -0,0 +1,78 @@
+STUNNEL + NFS QUICK REFERENCE FOR r1 AND r2
+===========================================
+
+COMPLETE SETUP (run as root on r1 and r2):
+------------------------------------------
+
+# 1. Install stunnel
+dnf install -y stunnel
+
+# 2. Copy certificate from f0 (run on f0)
+scp /usr/local/etc/stunnel/stunnel.pem root@r1:/etc/stunnel/
+scp /usr/local/etc/stunnel/stunnel.pem root@r2:/etc/stunnel/
+
+# 3. Create stunnel config on r1/r2
+mkdir -p /etc/stunnel
+cat > /etc/stunnel/stunnel.conf <<'EOF'
+cert = /etc/stunnel/stunnel.pem
+client = yes
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+EOF
+
+# 4. Create systemd service
+cat > /etc/systemd/system/stunnel.service <<'EOF'
+[Unit]
+Description=SSL tunnel for network daemons
+After=network.target
+
+[Service]
+Type=forking
+ExecStart=/usr/bin/stunnel /etc/stunnel/stunnel.conf
+ExecStop=/usr/bin/killall stunnel
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+# 5. Enable and start stunnel
+systemctl daemon-reload
+systemctl enable --now stunnel
+
+# 6. Create mount point
+mkdir -p /data/nfs/k3svolumes
+
+# 7. Test mount
+mount -t nfs4 -o port=2323 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes
+
+# 8. Verify mount works
+ls -la /data/nfs/k3svolumes/
+
+# 9. Add to fstab for persistence
+echo "127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0" >> /etc/fstab
+
+# 10. Test fstab mount
+umount /data/nfs/k3svolumes
+mount /data/nfs/k3svolumes
+
+VERIFICATION COMMANDS:
+----------------------
+systemctl status stunnel
+mount | grep k3svolumes
+df -h /data/nfs/k3svolumes
+echo "test" > /data/nfs/k3svolumes/test-$(hostname).txt
+
+TROUBLESHOOTING:
+----------------
+# Check stunnel logs
+journalctl -u stunnel -f
+
+# Test connectivity
+telnet 127.0.0.1 2323
+
+# Restart services
+systemctl restart stunnel
+umount /data/nfs/k3svolumes && mount /data/nfs/k3svolumes \ No newline at end of file