From 7c82d9c821a900e8970c8c1c6b1d85ec9bde734a Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 9 Jul 2025 00:54:06 +0300 Subject: Update content for html --- ...2025-07-01-posts-from-january-to-june-2025.html | 8 - .../DRAFT-f3s-kubernetes-with-freebsd-part-6.html | 457 +++++++++++++++++++-- gemfeed/STUNNEL-NFS-SETUP-R1-R2.md | 277 +++++++++++++ gemfeed/atom.xml | 12 +- gemfeed/configure-stunnel-nfs-r1-r2.sh | 163 ++++++++ gemfeed/stunnel-nfs-quick-reference.txt | 78 ++++ 6 files changed, 951 insertions(+), 44 deletions(-) create mode 100644 gemfeed/STUNNEL-NFS-SETUP-R1-R2.md create mode 100755 gemfeed/configure-stunnel-nfs-r1-r2.sh create mode 100644 gemfeed/stunnel-nfs-quick-reference.txt (limited to 'gemfeed') diff --git a/gemfeed/2025-07-01-posts-from-january-to-june-2025.html b/gemfeed/2025-07-01-posts-from-january-to-june-2025.html index 496b5c6e..d074fed1 100644 --- a/gemfeed/2025-07-01-posts-from-january-to-june-2025.html +++ b/gemfeed/2025-07-01-posts-from-january-to-june-2025.html @@ -39,7 +39,6 @@
  • ⇢ ⇢ I think discussing action items in incident ...
  • ⇢ ⇢ At first, functional options add a bit of ...
  • ⇢ ⇢ In the "Working with an SRE Interview" I have ...
    • -
  • ⇢ ⇢ In the "Working with an SRE Interview" I have ...
  • ⇢ ⇢ Small introduction to the #Android ...
  • ⇢ ⇢ Helix 2025.01 has been released. The completion ...
  • ⇢ ⇢ I found these are excellent examples of how ...
    • @@ -207,13 +206,6 @@ foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.gmi (Gemini)
    foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html

    -

    In the "Working with an SRE Interview" I have ...


    -
    -In the "Working with an SRE Interview" I have been askd about what it's like working with an SRE! We'd covered much more in depth, but we decided not to make it too long in the final version! #sre #interview
    -
    -foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.gmi (Gemini)
    -foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html
    -

    Small introduction to the #Android ...



    Small introduction to the #Android distribution called #GrapheneOS For myself, I am using a Pixel 7 Pro, which comes with "only" 5 years of support (not yet 7 years like the Pixel 8 and 9 series). I also wrote about GrapheneOS here once:
    diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html index f0a3800d..6d4f42d9 100644 --- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html +++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html @@ -37,6 +37,7 @@
  • ⇢ ⇢ Migrating Bhyve VMs to encrypted bhyve ZFS volume
  • ⇢ CARP
  • ⇢ ZFS Replication with zrepl
    • +
  • ⇢ ⇢ Why zrepl instead of HAST?
  • ⇢ ⇢ Installing zrepl
  • ⇢ ⇢ Checking ZFS pools
  • ⇢ ⇢ Configuring zrepl with WireGuard tunnel
    • @@ -45,6 +46,13 @@
  • ⇢ ⇢ Enabling and starting zrepl services
  • ⇢ ⇢ Verifying replication
  • ⇢ ⇢ Monitoring replication
    • +
  • ⇢ ⇢ A note about the Bhyve VM replication
    • +
  • ⇢ ⇢ Quick status check commands
    • +
  • ⇢ ⇢ Verifying replication after reboot
    • +
  • ⇢ ⇢ Important note about failover limitations
    • +
  • ⇢ ⇢ Mounting the NFS datasets
    • +
  • ⇢ ⇢ Failback scenario: Syncing changes from f1 back to f0
    • +
  • ⇢ ⇢ Testing the failback scenario

    • Introduction



    @@ -228,18 +236,31 @@ zroot/bhyve/rocky keystatus available -
    In this section, we'll set up automatic ZFS replication from f0 to f1 using zrepl. This ensures our data is replicated across nodes for redundancy.

    +

    Why zrepl instead of HAST?


    +
    +While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage, I've chosen zrepl for several important reasons:
    +
    +1. **HAST can cause ZFS corruption**: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.
    +
    +2. **ZFS-aware replication**: zrepl understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication.
    +
    +3. **Snapshot history**: With zrepl, you get multiple recovery points (every 5 minutes in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.
    +
    +4. **Easier recovery**: When something goes wrong with zrepl, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary too.
    +
    +5. **Network flexibility**: zrepl works over any TCP connection (in our case, WireGuard), while HAST requires dedicated network configuration.
    +
    +The 5-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database - it's storage for personal projects, development work, and home lab experiments. Losing at most 5 minutes of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication.
    +

    Installing zrepl



    First, install zrepl on both hosts:

    - -
    # On f0
+
    +# On f0
 paul@f0:~ % doas pkg install -y zrepl
 
-# On f1
+# On f1
 paul@f1:~ % doas pkg install -y zrepl
 
    
 
    
@@ -290,6 +311,16 @@ paul@f1:~ % ifconfig wg0 | grep inet
 
    
 
    Configuring zrepl on f0 (source)

    
 
    
+First, create a dedicated dataset for NFS data that will be replicated:
    
+
    
+
+
    # Create the nfsdata dataset that will hold all data exposed via NFS
+paul@f0:~ % doas zfs create zdata/enc/nfsdata
+
    
+
    
 Create the zrepl configuration on f0:
    
 
    
 
-
    paul@f1:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<'EOF'
+
    # First create a dedicated sink dataset
+paul@f1:~ % doas zfs create zdata/sink
+
+paul@f1:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<'EOF'
 global:
   logging:
     - type: stdout
@@ -358,7 +390,7 @@ global:
     recv:
       placeholder:
         encryption: inherit
-    root_fs: "zdata/enc"
+    root_fs: "zdata/sink"
 EOF
 
    
 
    
@@ -391,17 +423,31 @@ Starting zrepl.
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-
    # On f0, check zrepl status
-paul@f0:~ % doas zrepl status
+
    # On f0, check zrepl status (use raw mode for non-tty)
+paul@f0:~ % doas zrepl status --mode raw | grep -A2 "Replication"
+"Replication":{"StartAt":"2025-07-01T22:31:48.712143123+03:00"...
+
+# Check if services are running
+paul@f0:~ % doas service zrepl status
+zrepl is running as pid 2649.
+
+paul@f1:~ % doas service zrepl status
+zrepl is running as pid 2574.
 
-# Check for zrepl snapshots
+# Check for zrepl snapshots on source
 paul@f0:~ % doas zfs list -t snapshot -r zdata/enc | grep zrepl
+zdata/enc@zrepl_20250701_193148_000    0B      -   176K  -
 
-# On f1, verify the replicated datasets
-paul@f1:~ % doas zfs list -r zdata/enc
+# On f1, verify the replicated datasets  
+paul@f1:~ % doas zfs list -r zdata | grep f0
+zdata/f0             576K   899G   200K  none
+zdata/f0/zdata       376K   899G   200K  none
+zdata/f0/zdata/enc   176K   899G   176K  none
 
-# Check zrepl logs for any errors
-paul@f0:~ % doas tail -f /var/log/zrepl.log
+# Check replicated snapshots on f1
+paul@f1:~ % doas zfs list -t snapshot -r zdata | grep zrepl
+zdata/f0/zdata/enc@zrepl_20250701_193148_000     0B      -   176K  -
+zdata/f0/zdata/enc@zrepl_20250701_194148_000     0B      -   176K  -
 
    
 
    
 
    Monitoring replication

    
@@ -419,8 +465,365 @@ paul@f0:~ % doas zrepl status --mode interactive
 paul@f0:~ % doas zrepl status --job f0_to_f1
 
    
 
    
-With this setup, zdata/enc on f0 will be automatically replicated to f1 every 10 minutes, with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep recent snapshots while managing disk space efficiently.
    
+With this setup, both zdata/enc/nfsdata and zroot/bhyve/fedora on f0 will be automatically replicated to f1 every 5 minutes, with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep the last 10 snapshots while managing disk space efficiently.
    
+
    
+The replicated data appears on f1 under zdata/sink/ with the source host and dataset hierarchy preserved:
    
+
    
+
      
+
    • zdata/enc/nfsdata → zdata/sink/f0/zdata/enc/nfsdata
      • 
+
    • zroot/bhyve/fedora → zdata/sink/f0/zroot/bhyve/fedora
      • 
+

    
+This is by design - zrepl preserves the complete path from the source to ensure there are no conflicts when replicating from multiple sources. The replication uses the WireGuard tunnel for secure, encrypted transport between nodes.
    
+
    
+
    A note about the Bhyve VM replication

    
+
    
+While replicating a Bhyve VM (Fedora in this case) is slightly off-topic for the f3s series, I've included it here as it demonstrates zrepl's flexibility. This is a development VM I use occasionally to log in remotely for certain development tasks. Having it replicated ensures I have a backup copy available on f1 if needed.
    
+
    
+
    Quick status check commands

    
 
    
+Here are the essential commands to monitor replication status:
    
+
    
+
+
    # On the source node (f0) - check if replication is active
+paul@f0:~ % doas zrepl status --job f0_to_f1 | grep -E '(State|Last)'
+State: done
+LastError: 
+
+# List all zrepl snapshots on source
+paul@f0:~ % doas zfs list -t snapshot | grep zrepl
+zdata/enc/nfsdata@zrepl_20250701_202530_000             0B      -   200K  -
+zroot/bhyve/fedora@zrepl_20250701_202530_000            0B      -  2.97G  -
+
+# On the sink node (f1) - verify received datasets
+paul@f1:~ % doas zfs list -r zdata/sink
+NAME                                   USED  AVAIL  REFER  MOUNTPOINT
+zdata/sink                             3.0G   896G   200K  /data/sink
+zdata/sink/f0                          3.0G   896G   200K  none
+zdata/sink/f0/zdata                    472K   896G   200K  none
+zdata/sink/f0/zdata/enc                272K   896G   200K  none
+zdata/sink/f0/zdata/enc/nfsdata        176K   896G   176K  none
+zdata/sink/f0/zroot                    2.9G   896G   200K  none
+zdata/sink/f0/zroot/bhyve              2.9G   896G   200K  none
+zdata/sink/f0/zroot/bhyve/fedora       2.9G   896G  2.97G  none
+
+# Check received snapshots on sink
+paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep zrepl | wc -l
+       3
+
+# Monitor replication progress in real-time (on source)
+paul@f0:~ % doas zrepl status --mode interactive
+
+# Check last replication time (on source)
+paul@f0:~ % doas zrepl status --job f0_to_f1 | grep -A1 "Replication"
+Replication:
+  Status: Idle (last run: 2025-07-01T22:41:48)
+
+# View zrepl logs for troubleshooting
+paul@f0:~ % doas tail -20 /var/log/zrepl.log | grep -E '(error|warn|replication)'
+
    
+
    
+These commands provide a quick way to verify that:
    
+
    
+
      
+
    • Replication jobs are running without errors
      • 
+
    • Snapshots are being created on the source
      • 
+
    • Data is being received on the sink
      • 
+
    • The replication schedule is being followed
      • 
+

    
+
    Verifying replication after reboot

    
+
    
+The zrepl service is configured to start automatically at boot. After rebooting both hosts:
    
+
    
+
+
    paul@f0:~ % uptime
+11:17PM  up 1 min, 0 users, load averages: 0.16, 0.06, 0.02
+
+paul@f0:~ % doas service zrepl status
+zrepl is running as pid 2366.
+
+paul@f1:~ % doas service zrepl status
+zrepl is running as pid 2309.
+
+# Check that new snapshots are being created and replicated
+paul@f0:~ % doas zfs list -t snapshot | grep zrepl | tail -2
+zdata/enc/nfsdata@zrepl_20250701_202530_000                0B      -   200K  -
+zroot/bhyve/fedora@zrepl_20250701_202530_000               0B      -  2.97G  -
+
+paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep 202530
+zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_202530_000      0B      -   176K  -
+zdata/sink/f0/zroot/bhyve/fedora@zrepl_20250701_202530_000     0B      -  2.97G  -
+
    
+
    
+The timestamps confirm that replication resumed automatically after the reboot, ensuring continuous data protection.
    
+
    
+
    Important note about failover limitations

    
+
    
+The current zrepl setup provides **backup/disaster recovery** but not automatic failover. The replicated datasets on f1 are not mounted by default (mountpoint=none). In case f0 fails:
    
+
    
+
+
    # Manual steps needed on f1 to activate the replicated data:
+paul@f1:~ % doas zfs set mountpoint=/data/nfsdata zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
    
+
    
+However, this creates a **split-brain problem**: when f0 comes back online, both systems would have diverged data. Resolving this requires careful manual intervention to:
    
+
    
+1. Stop the original replication
    
+2. Sync changes from f1 back to f0
    
+3. Re-establish normal replication
    
+
    
+For true high-availability NFS, you might consider:
    
+
    
+
      
+
    • **Shared storage** (like iSCSI) with proper clustering
      • 
+
    • **GlusterFS** or similar distributed filesystems
      • 
+
    • **Manual failover with ZFS replication** (as we have here)
      • 
+

    
+Note: While HAST+CARP is often suggested for HA storage, it can cause filesystem corruption in practice, especially with ZFS. The block-level replication of HAST doesn't understand ZFS's transactional model, leading to inconsistent states during failover. 
    
+
    
+The current zrepl setup, despite requiring manual intervention, is actually safer because:
    
+
    
+
      
+
    • ZFS snapshots are always consistent
      • 
+
    • Replication is ZFS-aware (not just block-level)
      • 
+
    • You have full control over the failover process
      • 
+
    • No risk of split-brain corruption
      • 
+

    
+
    Mounting the NFS datasets

    
+
    
+To make the nfsdata accessible on both nodes, we need to mount them. On f0, this is straightforward:
    
+
    
+
+
    # On f0 - set mountpoint for the primary nfsdata
+paul@f0:~ % doas zfs set mountpoint=/data/nfs zdata/enc/nfsdata
+paul@f0:~ % doas mkdir -p /data/nfs
+
+# Verify it's mounted
+paul@f0:~ % df -h /data/nfs
+Filesystem           Size    Used   Avail Capacity  Mounted on
+zdata/enc/nfsdata    899G    204K    899G     0%    /data/nfs
+
    
+
    
+On f1, we need to handle the encryption key and mount the standby copy:
    
+
    
+
+
    # On f1 - first check encryption status
+paul@f1:~ % doas zfs get keystatus zdata/sink/f0/zdata/enc/nfsdata
+NAME                             PROPERTY   VALUE        SOURCE
+zdata/sink/f0/zdata/enc/nfsdata  keystatus  unavailable  -
+
+# Load the encryption key (using f0's key stored on the USB)
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
+    zdata/sink/f0/zdata/enc/nfsdata
+
+# Set mountpoint and mount (same path as f0 for easier failover)
+paul@f1:~ % doas mkdir -p /data/nfs
+paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
+# Make it read-only to prevent accidental writes that would break replication
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+
+# Verify
+paul@f1:~ % df -h /data/nfs
+Filesystem                         Size    Used   Avail Capacity  Mounted on
+zdata/sink/f0/zdata/enc/nfsdata    896G    204K    896G     0%    /data/nfs
+
    
+
    
+Note: The dataset is mounted at the same path (/data/nfs) on both hosts to simplify failover procedures. The dataset on f1 is set to readonly=on to prevent accidental modifications that would break replication.
    
+
    
+**CRITICAL WARNING**: Do NOT write to /data/nfs/ on f1! Any modifications will break the replication. If you accidentally write to it, you'll see this error:
    
+
    
+
    +cannot receive incremental stream: destination zdata/sink/f0/zdata/enc/nfsdata has been modified
+since most recent snapshot
+
    
+
    
+To fix a broken replication after accidental writes:
    
+
+
    # Option 1: Rollback to the last common snapshot (loses local changes)
+paul@f1:~ % doas zfs rollback zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_204054_000
+
+# Option 2: Make it read-only to prevent accidents
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+
    
+
    
+To ensure the encryption key is loaded automatically after reboot on f1:
    
+
+
    paul@f1:~ % doas sysrc zfskeys_datasets="zdata/sink/f0/zdata/enc/nfsdata"
+
    
+
    
+
    Failback scenario: Syncing changes from f1 back to f0

    
+
    
+In a disaster recovery scenario where f0 has failed and f1 has taken over, you'll need to sync changes back when f0 returns. Here's how to failback:
    
+
    
+
+
    # On f1: First, make the dataset writable (if it was readonly)
+paul@f1:~ % doas zfs set readonly=off zdata/sink/f0/zdata/enc/nfsdata
+
+# Create a snapshot of the current state
+paul@f1:~ % doas zfs snapshot zdata/sink/f0/zdata/enc/nfsdata@failback
+
+# On f0: Stop any services using the dataset
+paul@f0:~ % doas service nfsd stop  # If NFS is running
+
+# Send the snapshot from f1 to f0, forcing a rollback
+# This WILL DESTROY any data on f0 that's not on f1!
+paul@f1:~ % doas zfs send -R zdata/sink/f0/zdata/enc/nfsdata@failback | \
+    ssh f0 "doas zfs recv -F zdata/enc/nfsdata"
+
+# Alternative: If you want to see what would be received first
+paul@f1:~ % doas zfs send -R zdata/sink/f0/zdata/enc/nfsdata@failback | \
+    ssh f0 "doas zfs recv -nv -F zdata/enc/nfsdata"
+
+# After successful sync, on f0:
+paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@failback
+
+# On f1: Make it readonly again and destroy the failback snapshot
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@failback
+
+# Stop zrepl services first - CRITICAL!
+paul@f0:~ % doas service zrepl stop
+paul@f1:~ % doas service zrepl stop
+
+# Clean up any zrepl snapshots on f0
+paul@f0:~ % doas zfs list -t snapshot -r zdata/enc/nfsdata | grep zrepl | \
+    awk '{print $1}' | xargs -I {} doas zfs destroy {}
+
+# Clean up and destroy the entire replicated structure on f1
+# First release any holds
+paul@f1:~ % doas zfs holds -r zdata/sink/f0 | grep -v NAME | \
+    awk '{print $2, $1}' | while read tag snap; do 
+        doas zfs release "$tag" "$snap"
+    done
+
+# Then destroy the entire f0 tree
+paul@f1:~ % doas zfs destroy -rf zdata/sink/f0
+
+# Create parent dataset structure on f1
+paul@f1:~ % doas zfs create -p zdata/sink/f0/zdata/enc
+
+# Create a fresh manual snapshot to establish baseline
+paul@f0:~ % doas zfs snapshot zdata/enc/nfsdata@manual_baseline
+
+# Send this snapshot to f1
+paul@f0:~ % doas zfs send -w zdata/enc/nfsdata@manual_baseline | \
+    ssh f1 "doas zfs recv zdata/sink/f0/zdata/enc/nfsdata"
+
+# Clean up the manual snapshot
+paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@manual_baseline
+paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@manual_baseline
+
+# Set mountpoint and make readonly on f1
+paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+
+# Load encryption key and mount on f1
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
+    zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
+# Now restart zrepl services
+paul@f0:~ % doas service zrepl start
+paul@f1:~ % doas service zrepl start
+
+# Verify replication is working
+paul@f0:~ % doas zrepl status --job f0_to_f1
+
    
+
    
+**Important notes about failback**:
    
+
    
+
      
+
    • The -F flag forces a rollback on f0, destroying any local changes
      • 
+
    • Replication often won't resume automatically after a forced receive
      • 
+
    • You must clean up old zrepl snapshots on both sides
      • 
+
    • Creating a manual snapshot helps re-establish the replication relationship
      • 
+
    • Always verify replication status after the failback procedure
      • 
+
    • The first replication after failback will be a full send of the current state
      • 
+

    
+
    Testing the failback scenario

    
+
    
+Here's a real test of the failback procedure:
    
+
    
+
+
    # Simulate failure: Stop replication on f0
+paul@f0:~ % doas service zrepl stop
+
+# On f1: Take over by making the dataset writable
+paul@f1:~ % doas zfs set readonly=off zdata/sink/f0/zdata/enc/nfsdata
+
+# Write some data on f1 during the "outage"
+paul@f1:~ % echo 'Data written on f1 during failover' | doas tee /data/nfs/failover-data.txt
+Data written on f1 during failover
+
+# Now perform failback when f0 comes back online
+# Create snapshot on f1
+paul@f1:~ % doas zfs snapshot zdata/sink/f0/zdata/enc/nfsdata@failback
+
+# Send data back to f0 (note: we had to send to a temporary dataset due to holds)
+paul@f1:~ % doas zfs send -Rw zdata/sink/f0/zdata/enc/nfsdata@failback | \
+    ssh f0 "doas zfs recv -F zdata/enc/nfsdata_temp"
+
+# On f0: Rename datasets to complete failback
+paul@f0:~ % doas zfs set mountpoint=none zdata/enc/nfsdata
+paul@f0:~ % doas zfs rename zdata/enc/nfsdata zdata/enc/nfsdata_old
+paul@f0:~ % doas zfs rename zdata/enc/nfsdata_temp zdata/enc/nfsdata
+
+# Load encryption key and mount
+paul@f0:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
+paul@f0:~ % doas zfs mount zdata/enc/nfsdata
+
+# Verify the data from f1 is now on f0
+paul@f0:~ % ls -la /data/nfs/
+total 18
+drwxr-xr-x  2 root wheel  4 Jul  2 00:01 .
+drwxr-xr-x  4 root wheel  4 Jul  1 23:41 ..
+-rw-r--r--  1 root wheel 35 Jul  2 00:01 failover-data.txt
+-rw-r--r--  1 root wheel 12 Jul  1 23:34 hello.txt
+
    
+
    
+Success! The failover data from f1 is now on f0. To resume normal replication, you would need to:
    
+
    
+1. Clean up old snapshots on both sides
    
+2. Create a new manual baseline snapshot
    
+3. Restart zrepl services
    
+
    
+**Key learnings from the test**:
    
+
    
+
      
+
    • The -w flag is essential for encrypted datasets
      • 
+
    • Dataset holds can complicate the process (consider sending to a temporary dataset)
      • 
+
    • The encryption key must be loaded after receiving the dataset
      • 
+
    • Always verify data integrity before resuming normal operations
      • 
+

    
 ZFS auto scrubbing....~?
    
 
    
 Backup of the keys on the key locations (all keys on all 3 USB keys)
    
diff --git a/gemfeed/STUNNEL-NFS-SETUP-R1-R2.md b/gemfeed/STUNNEL-NFS-SETUP-R1-R2.md
new file mode 100644
index 00000000..3c2ff77f
--- /dev/null
+++ b/gemfeed/STUNNEL-NFS-SETUP-R1-R2.md
@@ -0,0 +1,277 @@
+# Stunnel and NFS Configuration for r1 and r2
+
+This document provides step-by-step instructions for configuring stunnel and NFS mounts on r1 and r2 Rocky Linux systems to connect to the f3s storage cluster.
+
+## Prerequisites
+
+- Root access on r1 and r2
+- Network connectivity to f0 (for copying the certificate)
+- Network connectivity to the CARP VIP (192.168.1.138)
+
+## Overview
+
+The configuration provides:
+- Encrypted NFS traffic using stunnel
+- Automatic failover via CARP VIP (192.168.1.138)
+- Persistent mounts across reboots
+- Access to /data/nfs/k3svolumes for Kubernetes storage
+
+## Configuration Steps
+
+### Step 1: Install stunnel
+
+```bash
+dnf install -y stunnel
+```
+
+### Step 2: Copy the stunnel certificate from f0
+
+First, create the directory:
+```bash
+mkdir -p /etc/stunnel
+```
+
+Then copy the certificate from f0. On f0, run:
+```bash
+scp /usr/local/etc/stunnel/stunnel.pem root@r1:/etc/stunnel/
+scp /usr/local/etc/stunnel/stunnel.pem root@r2:/etc/stunnel/
+```
+
+### Step 3: Create stunnel client configuration
+
+Create `/etc/stunnel/stunnel.conf`:
+```bash
+cat > /etc/stunnel/stunnel.conf <<'EOF'
+cert = /etc/stunnel/stunnel.pem
+client = yes
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+EOF
+```
+
+### Step 4: Create systemd service for stunnel
+
+Create `/etc/systemd/system/stunnel.service`:
+```bash
+cat > /etc/systemd/system/stunnel.service <<'EOF'
+[Unit]
+Description=SSL tunnel for network daemons
+After=network.target
+
+[Service]
+Type=forking
+ExecStart=/usr/bin/stunnel /etc/stunnel/stunnel.conf
+ExecStop=/usr/bin/killall stunnel
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+EOF
+```
+
+### Step 5: Enable and start stunnel
+
+```bash
+systemctl daemon-reload
+systemctl enable stunnel
+systemctl start stunnel
+systemctl status stunnel
+```
+
+### Step 6: Create NFS mount point
+
+```bash
+mkdir -p /data/nfs/k3svolumes
+```
+
+### Step 7: Test mount NFS through stunnel
+
+```bash
+mount -t nfs4 -o port=2323 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes
+```
+
+### Step 8: Verify the mount
+
+```bash
+mount | grep k3svolumes
+df -h /data/nfs/k3svolumes
+ls -la /data/nfs/k3svolumes/
+```
+
+### Step 9: Configure persistent mount
+
+First unmount the test mount:
+```bash
+umount /data/nfs/k3svolumes
+```
+
+Add to `/etc/fstab`:
+```bash
+echo "127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0" >> /etc/fstab
+```
+
+Mount using fstab:
+```bash
+mount /data/nfs/k3svolumes
+```
+
+## Automated Installation
+
+A script is available to automate all these steps:
+
+```bash
+# Download and run the configuration script
+curl -O https://raw.githubusercontent.com/.../configure-stunnel-nfs-r1-r2.sh
+chmod +x configure-stunnel-nfs-r1-r2.sh
+./configure-stunnel-nfs-r1-r2.sh
+```
+
+## Verification Commands
+
+After configuration, verify everything is working:
+
+```bash
+# Check stunnel service
+systemctl status stunnel
+
+# Check NFS mount
+mount | grep k3svolumes
+df -h /data/nfs/k3svolumes
+
+# Test write access
+echo "Test from $(hostname) at $(date)" > /data/nfs/k3svolumes/test-$(hostname).txt
+cat /data/nfs/k3svolumes/test-$(hostname).txt
+
+# Check stunnel connection
+ss -tlnp | grep 2323
+```
+
+## Troubleshooting
+
+### Stunnel won't start
+
+Check the logs:
+```bash
+journalctl -u stunnel -n 50
+```
+
+Common issues:
+- Certificate file missing or wrong permissions
+- Port 2323 already in use
+- Configuration syntax error
+
+### NFS mount fails
+
+Check connectivity:
+```bash
+# Test if stunnel is listening
+telnet 127.0.0.1 2323
+
+# Check if CARP VIP is reachable
+ping -c 3 192.168.1.138
+
+# Try mounting with verbose output
+mount -v -t nfs4 -o port=2323 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes
+```
+
+### Mount not persistent after reboot
+
+Verify fstab entry:
+```bash
+grep k3svolumes /etc/fstab
+```
+
+Test fstab mount:
+```bash
+mount -a
+```
+
+Check for systemd mount errors:
+```bash
+systemctl --failed
+journalctl -b | grep mount
+```
+
+### Permission denied errors
+
+The NFS export on f0/f1 maps root, so permission issues are rare. If they occur:
+```bash
+# Check export configuration on NFS server
+showmount -e 192.168.1.138
+
+# Verify your IP is allowed in the exports
+# r0: 192.168.1.120
+# r1: 192.168.1.121
+# r2: 192.168.1.122
+```
+
+## Security Considerations
+
+- All NFS traffic is encrypted through stunnel
+- The certificate provides both authentication and encryption
+- Access is restricted by IP address on the NFS server
+- Root access is mapped (maproot=root) for Kubernetes operations
+
+## Integration with Kubernetes
+
+Once configured, Kubernetes can use this mount for persistent storage:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: nfs-pv
+spec:
+  capacity:
+    storage: 10Gi
+  accessModes:
+    - ReadWriteMany
+  nfs:
+    server: 127.0.0.1  # Local stunnel
+    path: /data/nfs/k3svolumes
+  mountOptions:
+    - port=2323
+    - nfsvers=4
+```
+
+## Maintenance
+
+### Restarting services
+
+```bash
+# Restart stunnel
+systemctl restart stunnel
+
+# Remount NFS
+umount /data/nfs/k3svolumes
+mount /data/nfs/k3svolumes
+```
+
+### Updating certificates
+
+When certificates expire (after 10 years):
+1. Generate new certificate on f0
+2. Copy to all clients (r0, r1, r2)
+3. Restart stunnel on all hosts
+
+### Monitoring
+
+Add to your monitoring system:
+- stunnel service status
+- NFS mount presence
+- Disk space on /data/nfs/k3svolumes
+- Network connectivity to 192.168.1.138:2323
+
+## Summary
+
+After completing these steps on both r1 and r2:
+
+1. **Stunnel** provides encrypted tunnel to NFS server
+2. **NFS** mounts through stunnel on port 2323
+3. **CARP VIP** (192.168.1.138) ensures automatic failover
+4. **Persistent mount** via /etc/fstab survives reboots
+5. **Kubernetes** can use /data/nfs/k3svolumes for persistent volumes
+
+The same configuration works on r0, r1, and r2 with no modifications needed.
\ No newline at end of file
diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml
index 0b9cc97e..b2deaa44 100644
--- a/gemfeed/atom.xml
+++ b/gemfeed/atom.xml
@@ -1,6 +1,6 @@
 
 
-    2025-07-01T22:39:30+03:00
+    2025-07-02T00:37:08+03:00
     foo.zone feed
     To be in the .zone!
     
@@ -20,6 +20,8 @@
             
    
                 
    Posts from January to June 2025

    
 
    
+Published at 2025-07-01T22:39:29+03:00
    
+
    
 These are my social media posts from the last six months. I keep them here to reflect on them and also to not lose them. Social media networks come and go and are not under my control, but my domain is here to stay. 
    
 
    
 These are from Mastodon and LinkedIn. Have a look at my about page for my social media profiles. This list is generated with Gos, my social media platform sharing tool.
    
@@ -44,7 +46,6 @@
 
  • ⇢ ⇢ I think discussing action items in incident ...
    • 
 
  • ⇢ ⇢ At first, functional options add a bit of ...
    • 
 
  • ⇢ ⇢ In the "Working with an SRE Interview" I have ...
    • 
-
  • ⇢ ⇢ In the "Working with an SRE Interview" I have ...
    • 
 
  • ⇢ ⇢ Small introduction to the #Android ...
    • 
 
  • ⇢ ⇢ Helix 2025.01 has been released. The completion ...
    • 
 
  • ⇢ ⇢ I found these are excellent examples of how ...
    • 
@@ -212,13 +213,6 @@
 foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html (Gemini)
    
 foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html
    
 
    
-
    In the "Working with an SRE Interview" I have ...

    
-
    
-In the "Working with an SRE Interview" I have been askd about what it's like working with an SRE! We'd covered much more in depth, but we decided not to make it too long in the final version! #sre #interview
    
-
    
-foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html (Gemini)
    
-foo.zone/gemfeed/2025-01-15-working-with-an-sre-interview.html
    
-
    
 
    Small introduction to the #Android ...

    
 
    
 Small introduction to the #Android distribution called #GrapheneOS For myself, I am using a Pixel 7 Pro, which comes with "only" 5 years of support (not yet 7 years like the Pixel 8 and 9 series). I also wrote about GrapheneOS here once:
    
diff --git a/gemfeed/configure-stunnel-nfs-r1-r2.sh b/gemfeed/configure-stunnel-nfs-r1-r2.sh
new file mode 100755
index 00000000..58431744
--- /dev/null
+++ b/gemfeed/configure-stunnel-nfs-r1-r2.sh
@@ -0,0 +1,163 @@
+#!/bin/sh
+# Configure stunnel and NFS mounts on r1 and r2 for f3s storage
+# This script should be run on both r1 and r2 Rocky Linux systems
+
+set -e
+
+# Function to print colored status messages
+print_status() {
+    echo -e "\n\033[1;34m>>> $1\033[0m"
+}
+
+print_error() {
+    echo -e "\033[1;31mERROR: $1\033[0m"
+    exit 1
+}
+
+print_success() {
+    echo -e "\033[1;32m✓ $1\033[0m"
+}
+
+# Check if running as root
+if [ "$EUID" -ne 0 ]; then 
+    print_error "Please run as root"
+fi
+
+# Detect hostname
+HOSTNAME=$(hostname -s)
+if [ "$HOSTNAME" != "r1" ] && [ "$HOSTNAME" != "r2" ]; then
+    print_error "This script should only be run on r1 or r2"
+fi
+
+print_status "Configuring stunnel and NFS on $HOSTNAME"
+
+# Step 1: Install stunnel package
+print_status "Installing stunnel package..."
+dnf install -y stunnel || print_error "Failed to install stunnel"
+print_success "stunnel installed"
+
+# Step 2: Create stunnel directory
+print_status "Creating stunnel configuration directory..."
+mkdir -p /etc/stunnel
+print_success "Directory created"
+
+# Step 3: Copy stunnel certificate from f0
+print_status "Copying stunnel certificate from f0..."
+echo "Please copy the certificate manually:"
+echo "On f0, run: scp /usr/local/etc/stunnel/stunnel.pem root@$HOSTNAME:/etc/stunnel/"
+echo ""
+echo "Press Enter when the certificate has been copied..."
+read -r
+
+# Verify certificate exists
+if [ ! -f /etc/stunnel/stunnel.pem ]; then
+    print_error "Certificate not found at /etc/stunnel/stunnel.pem"
+fi
+print_success "Certificate found"
+
+# Step 4: Create stunnel client configuration
+print_status "Creating stunnel client configuration..."
+cat > /etc/stunnel/stunnel.conf <<'EOF'
+cert = /etc/stunnel/stunnel.pem
+client = yes
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+EOF
+print_success "Stunnel configuration created"
+
+# Step 5: Create systemd service for stunnel
+print_status "Creating stunnel systemd service..."
+cat > /etc/systemd/system/stunnel.service <<'EOF'
+[Unit]
+Description=SSL tunnel for network daemons
+After=network.target
+
+[Service]
+Type=forking
+ExecStart=/usr/bin/stunnel /etc/stunnel/stunnel.conf
+ExecStop=/usr/bin/killall stunnel
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+EOF
+print_success "Systemd service created"
+
+# Step 6: Enable and start stunnel service
+print_status "Enabling and starting stunnel service..."
+systemctl daemon-reload
+systemctl enable stunnel
+systemctl start stunnel
+systemctl status stunnel --no-pager || print_error "Stunnel failed to start"
+print_success "Stunnel service is running"
+
+# Step 7: Create mount point for NFS
+print_status "Creating NFS mount point..."
+mkdir -p /data/nfs/k3svolumes
+print_success "Mount point created"
+
+# Step 8: Test mount NFS through stunnel
+print_status "Testing NFS mount through stunnel..."
+mount -t nfs4 -o port=2323 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes || print_error "Failed to mount NFS"
+print_success "NFS mounted successfully"
+
+# Step 9: Verify the mount
+print_status "Verifying NFS mount..."
+mount | grep k3svolumes || print_error "NFS mount not found"
+df -h /data/nfs/k3svolumes
+print_success "NFS mount verified"
+
+# Step 10: Unmount for fstab configuration
+print_status "Unmounting NFS to configure fstab..."
+umount /data/nfs/k3svolumes
+print_success "Unmounted"
+
+# Step 11: Add entry to /etc/fstab for persistent mount
+print_status "Adding NFS mount to /etc/fstab..."
+# Check if entry already exists
+if grep -q "127.0.0.1:/data/nfs/k3svolumes" /etc/fstab; then
+    print_status "Entry already exists in /etc/fstab"
+else
+    echo "127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0" >> /etc/fstab
+    print_success "Added to /etc/fstab"
+fi
+
+# Step 12: Mount using fstab
+print_status "Mounting NFS using fstab entry..."
+mount /data/nfs/k3svolumes || print_error "Failed to mount from fstab"
+print_success "Mounted from fstab"
+
+# Step 13: Final verification
+print_status "Final verification..."
+echo ""
+echo "Stunnel status:"
+systemctl is-active stunnel || print_error "Stunnel is not active"
+echo ""
+echo "NFS mount status:"
+mount | grep k3svolumes
+echo ""
+echo "Disk usage:"
+df -h /data/nfs/k3svolumes
+echo ""
+
+# Step 14: Test write access
+print_status "Testing write access..."
+TEST_FILE="/data/nfs/k3svolumes/test-$HOSTNAME-$(date +%s).txt"
+echo "Test from $HOSTNAME at $(date)" > "$TEST_FILE" || print_error "Failed to write test file"
+cat "$TEST_FILE"
+print_success "Write test successful"
+
+print_success "Configuration complete on $HOSTNAME!"
+echo ""
+echo "Summary:"
+echo "- Stunnel is configured as a client connecting to 192.168.1.138:2323 (CARP VIP)"
+echo "- NFS is mounted through stunnel at /data/nfs/k3svolumes"
+echo "- Mount is persistent across reboots (configured in /etc/fstab)"
+echo "- All traffic between this host and the NFS server is encrypted"
+echo ""
+echo "To verify everything after reboot:"
+echo "  systemctl status stunnel"
+echo "  mount | grep k3svolumes"
+echo "  ls -la /data/nfs/k3svolumes/"
\ No newline at end of file
diff --git a/gemfeed/stunnel-nfs-quick-reference.txt b/gemfeed/stunnel-nfs-quick-reference.txt
new file mode 100644
index 00000000..ca7f577a
--- /dev/null
+++ b/gemfeed/stunnel-nfs-quick-reference.txt
@@ -0,0 +1,78 @@
+STUNNEL + NFS QUICK REFERENCE FOR r1 AND r2
+===========================================
+
+COMPLETE SETUP (run as root on r1 and r2):
+------------------------------------------
+
+# 1. Install stunnel
+dnf install -y stunnel
+
+# 2. Copy certificate from f0 (run on f0)
+scp /usr/local/etc/stunnel/stunnel.pem root@r1:/etc/stunnel/
+scp /usr/local/etc/stunnel/stunnel.pem root@r2:/etc/stunnel/
+
+# 3. Create stunnel config on r1/r2
+mkdir -p /etc/stunnel
+cat > /etc/stunnel/stunnel.conf <<'EOF'
+cert = /etc/stunnel/stunnel.pem
+client = yes
+
+[nfs-ha]
+accept = 127.0.0.1:2323
+connect = 192.168.1.138:2323
+EOF
+
+# 4. Create systemd service
+cat > /etc/systemd/system/stunnel.service <<'EOF'
+[Unit]
+Description=SSL tunnel for network daemons
+After=network.target
+
+[Service]
+Type=forking
+ExecStart=/usr/bin/stunnel /etc/stunnel/stunnel.conf
+ExecStop=/usr/bin/killall stunnel
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+# 5. Enable and start stunnel
+systemctl daemon-reload
+systemctl enable --now stunnel
+
+# 6. Create mount point
+mkdir -p /data/nfs/k3svolumes
+
+# 7. Test mount
+mount -t nfs4 -o port=2323 127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes
+
+# 8. Verify mount works
+ls -la /data/nfs/k3svolumes/
+
+# 9. Add to fstab for persistence
+echo "127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0" >> /etc/fstab
+
+# 10. Test fstab mount
+umount /data/nfs/k3svolumes
+mount /data/nfs/k3svolumes
+
+VERIFICATION COMMANDS:
+----------------------
+systemctl status stunnel
+mount | grep k3svolumes
+df -h /data/nfs/k3svolumes
+echo "test" > /data/nfs/k3svolumes/test-$(hostname).txt
+
+TROUBLESHOOTING:
+----------------
+# Check stunnel logs
+journalctl -u stunnel -f
+
+# Test connectivity
+telnet 127.0.0.1 2323
+
+# Restart services
+systemctl restart stunnel
+umount /data/nfs/k3svolumes && mount /data/nfs/k3svolumes
\ No newline at end of file
-- 
cgit v1.2.3