From 7f603973b37cec6dc67ff2fdc67ae40d50ed6894 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sun, 12 Oct 2025 11:11:15 +0300 Subject: Update content for gemtext --- gemfeed/atom.xml | 165 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 163 insertions(+), 2 deletions(-) (limited to 'gemfeed/atom.xml') diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 1c38897c..e4df32dc 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ - 2025-10-11T15:44:10+03:00 + 2025-10-12T11:09:49+03:00 foo.zone feed To be in the .zone! @@ -265,7 +265,7 @@ p hash.values_at(:a, :c)
Other Ruby-related posts:

-2025-10-11 Key Takeaways from The Well-Grounded Rubyist (You are currently reading this)
+2025-10-11 Key kakeaways from The Well-Grounded Rubyist (You are currently reading this)
2021-07-04 The Well-Grounded Rubyist

Back to the main site
@@ -2664,6 +2664,13 @@ content = "{CODE}"
  • ⇢ ⇢ Mounting the NFS datasets
  • ⇢ ⇢ Troubleshooting: Files not appearing in replication
  • ⇢ ⇢ Configuring automatic key loading on boot
    • +
  • ⇢ ⇢ Troubleshooting: zrepl Replication Not Working
    • +
  • ⇢ ⇢# Check if zrepl Services are Running
    • +
  • ⇢ ⇢# Check zrepl Status for Errors
    • +
  • ⇢ ⇢# Fixing "No Common Snapshot" Errors
    • +
  • ⇢ ⇢# Network Connectivity Issues
    • +
  • ⇢ ⇢# Encryption Key Issues
    • +
  • ⇢ ⇢# Monitoring Ongoing Replication
  • ⇢ CARP (Common Address Redundancy Protocol)
  • ⇢ ⇢ How CARP Works
  • ⇢ ⇢ Configuring CARP
    • @@ -3402,6 +3409,160 @@ paul@f1:~ % doas zfs set \
  • Always verify datasets are mounted after reboot with zfs list -o name,mounted
  • Critical: Always ensure the replicated dataset on f1 remains read-only with doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata

    • +

    Troubleshooting: zrepl Replication Not Working


    +
    +If zrepl replication is not working, here's a systematic approach to diagnose and fix common issues:
    +
    +#### Check if zrepl Services are Running
    +
    +First, verify that zrepl is running on both nodes:
    +
    + +
    # Check service status on both f0 and f1
+paul@f0:~ % doas service zrepl status
+paul@f1:~ % doas service zrepl status
+
+# If not running, start the service
+paul@f0:~ % doas service zrepl start
+paul@f1:~ % doas service zrepl start
+
    +
    +#### Check zrepl Status for Errors
    +
    +Use the status command to see detailed error information:
    +
    + +
    # Check detailed status (use --mode raw for non-tty environments)
+paul@f0:~ % doas zrepl status --mode raw
+
+# Look for error messages in the replication section
+# Common errors include "no common snapshot" or connection failures
+
    +
    +#### Fixing "No Common Snapshot" Errors
    +
    +This is the most common replication issue, typically occurring when:
    +
    +
    +**Error message example:**
    +
    +no common snapshot or suitable bookmark between sender and receiver
+
    +
    +**Solution: Clean up conflicting snapshots on receiver**
    +
    + +
    # First, identify the destination dataset on f1
+paul@f1:~ % doas zfs list | grep sink
+
+# Check existing snapshots on the problematic dataset
+paul@f1:~ % doas zfs list -t snapshot | grep nfsdata
+
+# If you see snapshots with different naming (e.g., @daily-*, @weekly-*)
+# these conflict with zrepl's @zrepl_* snapshots
+
+# Destroy the entire destination dataset to allow clean replication
+paul@f1:~ % doas zfs destroy -r zdata/sink/f0/zdata/enc/nfsdata
+
+# For VM replication, do the same for the fedora dataset
+paul@f1:~ % doas zfs destroy -r zdata/sink/f0/zroot/bhyve/fedora
+
+# Wake up zrepl to start fresh replication
+paul@f0:~ % doas zrepl signal wakeup f0_to_f1_nfsdata
+paul@f0:~ % doas zrepl signal wakeup f0_to_f1_fedora
+
+# Check replication status
+paul@f0:~ % doas zrepl status --mode raw
+
    +
    +**Verification that replication is working:**
    +
    + +
    # Look for "stepping" state and active zfs send processes
+paul@f0:~ % doas zrepl status --mode raw | grep -A5 "State.*stepping"
+
+# Check for active ZFS commands
+paul@f0:~ % doas zrepl status --mode raw | grep -A10 "ZFSCmds.*Active"
+
+# Monitor progress - bytes replicated should be increasing
+paul@f0:~ % doas zrepl status --mode raw | grep BytesReplicated
+
    +
    +#### Network Connectivity Issues
    +
    +If replication fails to connect:
    +
    + +
    # Test connectivity between nodes
+paul@f0:~ % nc -zv 192.168.2.131 8888
+
+# Check if zrepl is listening on f1
+paul@f1:~ % doas netstat -an | grep 8888
+
+# Verify WireGuard tunnel is working
+paul@f0:~ % ping 192.168.2.131
+
    +
    +#### Encryption Key Issues
    +
    +If encrypted replication fails:
    +
    + +
    # Verify encryption keys are available on both nodes
+paul@f0:~ % doas zfs get keystatus zdata/enc/nfsdata
+paul@f1:~ % doas zfs get keystatus zdata/sink/f0/zdata/enc/nfsdata
+
+# Load keys if unavailable
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
+    zdata/sink/f0/zdata/enc/nfsdata
+
    +
    +#### Monitoring Ongoing Replication
    +
    +After fixing issues, monitor replication health:
    +
    + +
    # Monitor replication progress (run repeatedly to check status)
+paul@f0:~ % doas zrepl status --mode raw | grep -A10 BytesReplicated
+
+# Or install watch from ports and use it
+paul@f0:~ % doas pkg install watch
+paul@f0:~ % watch -n 5 'doas zrepl status --mode raw | grep -A10 BytesReplicated'
+
+# Check for new snapshots being created
+paul@f0:~ % doas zfs list -t snapshot | grep zrepl | tail -5
+
+# Verify snapshots appear on receiver
+paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep zrepl | tail -5
+
    +
    +This troubleshooting process resolves the most common zrepl issues and ensures continuous data replication between your storage nodes.
    +

    CARP (Common Address Redundancy Protocol)



    High availability is crucial for storage systems. If the storage server goes down, all NFS clients (which will also be Kubernetes pods later on in this series) lose access to their persistent data. CARP provides a solution by creating a virtual IP address that automatically migrates to a different server during failures. This means that clients point to that VIP for NFS mounts and are always contacting the current primary node.
    -- cgit v1.2.3