1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>f3s: Kubernetes with FreeBSD - Part 6: Storage</title>
<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
<link rel="stylesheet" href="../style.css" />
<link rel="stylesheet" href="style-override.css" />
</head>
<body>
<p class="header">
<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi">Gemini</a>
</p>
<h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-6-storage'>f3s: Kubernetes with FreeBSD - Part 6: Storage</h1><br />
<br />
<span class='quote'>Published at 2025-04-04T23:21:01+03:00</span><br />
<br />
<span>This is the sixth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.</span><br />
<br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<br />
<a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br />
<br />
<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
<br />
<ul>
<li><a href='#f3s-kubernetes-with-freebsd---part-6-storage'>f3s: Kubernetes with FreeBSD - Part 6: Storage</a></li>
<li>⇢ <a href='#introduction'>Introduction</a></li>
<li>⇢ <a href='#zfs-encryption-keys'>ZFS encryption keys</a></li>
<li>⇢ ⇢ <a href='#ufs-on-usb-keys'>UFS on USB keys</a></li>
<li>⇢ ⇢ <a href='#generating-encryption-keys'>Generating encryption keys</a></li>
<li>⇢ ⇢ <a href='#configuring-zdata-zfs-pool-and-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool and encryption</a></li>
<li>⇢ ⇢ <a href='#migrating-bhyve-vms-to-encrypted-bhyve-zfs-volume'>Migrating Bhyve VMs to encrypted <span class='inlinecode'>bhyve</span> ZFS volume</a></li>
<li>⇢ <a href='#carp'>CARP</a></li>
<li>⇢ <a href='#zfs-replication-with-zrepl'>ZFS Replication with zrepl</a></li>
<li>⇢ ⇢ <a href='#why-zrepl-instead-of-hast'>Why zrepl instead of HAST?</a></li>
<li>⇢ ⇢ <a href='#installing-zrepl'>Installing zrepl</a></li>
<li>⇢ ⇢ <a href='#checking-zfs-pools'>Checking ZFS pools</a></li>
<li>⇢ ⇢ <a href='#configuring-zrepl-with-wireguard-tunnel'>Configuring zrepl with WireGuard tunnel</a></li>
<li>⇢ ⇢ <a href='#configuring-zrepl-on-f0-source'>Configuring zrepl on f0 (source)</a></li>
<li>⇢ ⇢ <a href='#configuring-zrepl-on-f1-sink'>Configuring zrepl on f1 (sink)</a></li>
<li>⇢ ⇢ <a href='#enabling-and-starting-zrepl-services'>Enabling and starting zrepl services</a></li>
<li>⇢ ⇢ <a href='#verifying-replication'>Verifying replication</a></li>
<li>⇢ ⇢ <a href='#monitoring-replication'>Monitoring replication</a></li>
<li>⇢ ⇢ <a href='#a-note-about-the-bhyve-vm-replication'>A note about the Bhyve VM replication</a></li>
<li>⇢ ⇢ <a href='#quick-status-check-commands'>Quick status check commands</a></li>
<li>⇢ ⇢ <a href='#verifying-replication-after-reboot'>Verifying replication after reboot</a></li>
<li>⇢ ⇢ <a href='#important-note-about-failover-limitations'>Important note about failover limitations</a></li>
<li>⇢ ⇢ <a href='#mounting-the-nfs-datasets'>Mounting the NFS datasets</a></li>
<li>⇢ ⇢ <a href='#failback-scenario-syncing-changes-from-f1-back-to-f0'>Failback scenario: Syncing changes from f1 back to f0</a></li>
<li>⇢ ⇢ <a href='#testing-the-failback-scenario'>Testing the failback scenario</a></li>
</ul><br />
<h2 style='display: inline' id='introduction'>Introduction</h2><br />
<br />
<span>In this blog post, we are going to extend the Beelinks with some additional storage.</span><br />
<br />
<span>Some photos here, describe why there are 2 different models of SSD drives (replication etc)</span><br />
<br />
<h2 style='display: inline' id='zfs-encryption-keys'>ZFS encryption keys</h2><br />
<br />
<h3 style='display: inline' id='ufs-on-usb-keys'>UFS on USB keys</h3><br />
<br />
<pre>
paul@f0:/ % doas camcontrol devlist
<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0)
<Samsung SSD 870 EVO 1TB SVT03B6Q> at scbus1 target 0 lun 0 (pass1,ada1)
<Generic Flash Disk 8.07> at scbus2 target 0 lun 0 (da0,pass2)
paul@f0:/ %
</pre>
<br />
<pre>
paul@f1:/ % doas camcontrol devlist
<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0)
<CT1000BX500SSD1 M6CR072> at scbus1 target 0 lun 0 (pass1,ada1)
<Generic Flash Disk 8.07> at scbus2 target 0 lun 0 (da0,pass2)
paul@f1:/ %
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:/ % doas newfs /dev/da<font color="#000000">0</font>
/dev/da<font color="#000000">0</font>: <font color="#000000">15000</font>.0MB (<font color="#000000">30720000</font> sectors) block size <font color="#000000">32768</font>, fragment size <font color="#000000">4096</font>
using <font color="#000000">24</font> cylinder groups of <font color="#000000">625</font>.22MB, <font color="#000000">20007</font> blks, <font color="#000000">80128</font> inodes.
with soft updates
super-block backups (<b><u><font color="#000000">for</font></u></b> fsck_ffs -b <i><font color="silver">#) at:</font></i>
<font color="#000000">192</font>, <font color="#000000">1280640</font>, <font color="#000000">2561088</font>, <font color="#000000">3841536</font>, <font color="#000000">5121984</font>, <font color="#000000">6402432</font>, <font color="#000000">7682880</font>, <font color="#000000">8963328</font>, <font color="#000000">10243776</font>,
<font color="#000000">11524224</font>, <font color="#000000">12804672</font>, <font color="#000000">14085120</font>, <font color="#000000">15365568</font>, <font color="#000000">16646016</font>, <font color="#000000">17926464</font>, <font color="#000000">19206912</font>,k <font color="#000000">20487360</font>,
...
paul@f0:/ % echo <font color="#808080">'/dev/da0 /keys ufs rw 0 2'</font> | doas tee -a /etc/fstab
/dev/da<font color="#000000">0</font> /keys ufs rw <font color="#000000">0</font> <font color="#000000">2</font>
paul@f0:/ % doas mkdir /keys
paul@f0:/ % doas mount /keys
paul@f0:/ % df | grep keys
/dev/da<font color="#000000">0</font> <font color="#000000">14877596</font> <font color="#000000">8</font> <font color="#000000">13687384</font> <font color="#000000">0</font>% /keys
</pre>
<br />
<h3 style='display: inline' id='generating-encryption-keys'>Generating encryption keys</h3><br />
<br />
<span>paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32</span><br />
<span>paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:bhyve.key 32</span><br />
<span>paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:bhyve.key 32</span><br />
<span>paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:zdata.key 32</span><br />
<span>paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:zdata.key 32</span><br />
<span>paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:zdata.key 32</span><br />
<span>paul@f0:/keys % doas chown root *</span><br />
<span>paul@f0:/keys % doas chmod 400 *</span><br />
<br />
<span>paul@f0:/keys % ls -l</span><br />
<span>total 20</span><br />
<span>-r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key</span><br />
<span>-r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key</span><br />
<span>-r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key</span><br />
<span>-r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key</span><br />
<span>-r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key</span><br />
<span>-r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key</span><br />
<br />
<span>Copy those to all 3 nodes to /keys</span><br />
<br />
<h3 style='display: inline' id='configuring-zdata-zfs-pool-and-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool and encryption</h3><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:/keys % doas zpool create -m /data zdata /dev/ada<font color="#000000">1</font>
paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:zdata.key zdata/enc
paul@f0:/ % zfs list | grep zdata
zdata 836K 899G 96K /data
zdata/enc 200K 899G 200K /data/enc
paul@f0:/keys % zfs get all zdata/enc | grep -E -i <font color="#808080">'(encryption|key)'</font>
zdata/enc encryption aes-<font color="#000000">256</font>-gcm -
zdata/enc keylocation file:///keys/f<font color="#000000">0</font>.lan.buetow.org:zdata.key <b><u><font color="#000000">local</font></u></b>
zdata/enc keyformat raw -
zdata/enc encryptionroot zdata/enc -
zdata/enc keystatus available -
</pre>
<br />
<h3 style='display: inline' id='migrating-bhyve-vms-to-encrypted-bhyve-zfs-volume'>Migrating Bhyve VMs to encrypted <span class='inlinecode'>bhyve</span> ZFS volume</h3><br />
<br />
<span>Run on all 3 nodes</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:/keys % doas vm stop rocky
Sending ACPI shutdown to rocky
paul@f0:/keys % doas vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
rocky default uefi <font color="#000000">4</font> 14G - Yes [<font color="#000000">1</font>] Stopped
paul@f0:/keys % doas zfs rename zroot/bhyve zroot/bhyve_old
paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/mnt zroot/bhyve_old
paul@f0:/keys % doas zfs snapshot zroot/bhyve_old/rocky@hamburger
paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:bhyve.key zroot/bhyve
paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/zroot/bhyve zroot/bhyve
paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/zroot/bhyve/rocky zroot/bhyve/rocky
paul@f0:/keys % doas zfs send zroot/bhyve_old/rocky@hamburger | doas zfs recv zroot/bhyve/rocky
paul@f0:/keys % doas cp -Rp /mnt/.config /zroot/bhyve/
paul@f0:/keys % doas cp -Rp /mnt/.img /zroot/bhyve/
paul@f0:/keys % doas cp -Rp /mnt/.templates /zroot/bhyve/
paul@f0:/keys % doas cp -Rp /mnt/.iso /zroot/bhyve/
paul@f0:/keys % doas sysrc zfskeys_enable=YES
zfskeys_enable: -> YES
paul@f0:/keys % doas vm init
paul@f0:/keys % doas reboot
.
.
.
paul@f0:~ % doas vm list
paul@f0:~ % doas vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
rocky default uefi <font color="#000000">4</font> 14G <font color="#000000">0.0</font>.<font color="#000000">0.0</font>:<font color="#000000">5900</font> Yes [<font color="#000000">1</font>] Running (<font color="#000000">2265</font>)
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas zfs destroy -R zroot/bhyve_old
paul@f0:~ % zfs get all zroot/bhyve | grep -E <font color="#808080">'(encryption|key)'</font>
zroot/bhyve encryption aes-<font color="#000000">256</font>-gcm -
zroot/bhyve keylocation file:///keys/f<font color="#000000">0</font>.lan.buetow.org:bhyve.key <b><u><font color="#000000">local</font></u></b>
zroot/bhyve keyformat raw -
zroot/bhyve encryptionroot zroot/bhyve -
zroot/bhyve keystatus available -
paul@f0:~ % zfs get all zroot/bhyve/rocky | grep -E <font color="#808080">'(encryption|key)'</font>
zroot/bhyve/rocky encryption aes-<font color="#000000">256</font>-gcm -
zroot/bhyve/rocky keylocation none default
zroot/bhyve/rocky keyformat raw -
zroot/bhyve/rocky encryptionroot zroot/bhyve -
zroot/bhyve/rocky keystatus available -
</pre>
<br />
<h2 style='display: inline' id='carp'>CARP</h2><br />
<br />
<span>adding to /etc/rc.conf on f0 and f1:</span><br />
<span>ifconfig_re0_alias0="inet vhid 1 pass testpass alias 192.168.1.138/32"</span><br />
<br />
<span>adding to /etc/hosts:</span><br />
<br />
<span>192.168.1.138 f3s-storage-ha f3s-storage-ha.lan f3s-storage-ha.lan.buetow.org</span><br />
<br />
<span>Adding on f0 and f1:</span><br />
<br />
<span>paul@f0:~ % cat <<END | doas tee -a /etc/devd.conf</span><br />
<span>notify 0 {</span><br />
<span> match "system" "CARP";</span><br />
<span> match "subsystem" "[0-9]+@[0-9a-z.]+";</span><br />
<span> match "type" "(MASTER|BACKUP)";</span><br />
<span> action "/usr/local/bin/carpcontrol.sh $subsystem $type";</span><br />
<span>};</span><br />
<span>END</span><br />
<br />
<span>next, copied that script /usr/local/bin/carpcontrol.sh and adjusted the disk to storage</span><br />
<br />
<span>/boot/loader.conf add carp_load="YES"</span><br />
<span>reboot or run doas kldload carp0 </span><br />
<br />
<br />
<h2 style='display: inline' id='zfs-replication-with-zrepl'>ZFS Replication with zrepl</h2><br />
<br />
<span>In this section, we'll set up automatic ZFS replication from f0 to f1 using zrepl. This ensures our data is replicated across nodes for redundancy.</span><br />
<br />
<h3 style='display: inline' id='why-zrepl-instead-of-hast'>Why zrepl instead of HAST?</h3><br />
<br />
<span>While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage, I've chosen zrepl for several important reasons:</span><br />
<br />
<span>1. **HAST can cause ZFS corruption**: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.</span><br />
<br />
<span>2. **ZFS-aware replication**: zrepl understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication.</span><br />
<br />
<span>3. **Snapshot history**: With zrepl, you get multiple recovery points (every 5 minutes in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.</span><br />
<br />
<span>4. **Easier recovery**: When something goes wrong with zrepl, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary too.</span><br />
<br />
<span>5. **Network flexibility**: zrepl works over any TCP connection (in our case, WireGuard), while HAST requires dedicated network configuration.</span><br />
<br />
<span>The 5-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database - it's storage for personal projects, development work, and home lab experiments. Losing at most 5 minutes of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication.</span><br />
<br />
<h3 style='display: inline' id='installing-zrepl'>Installing zrepl</h3><br />
<br />
<span>First, install zrepl on both hosts:</span><br />
<br />
<pre>
# On f0
paul@f0:~ % doas pkg install -y zrepl
# On f1
paul@f1:~ % doas pkg install -y zrepl
</pre>
<br />
<h3 style='display: inline' id='checking-zfs-pools'>Checking ZFS pools</h3><br />
<br />
<span>Verify the pools and datasets on both hosts:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># On f0</font></i>
paul@f0:~ % doas zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zdata 928G <font color="#000000">1</font>.03M 928G - - <font color="#000000">0</font>% <font color="#000000">0</font>% <font color="#000000">1</font>.00x ONLINE -
zroot 472G <font color="#000000">26</font>.7G 445G - - <font color="#000000">0</font>% <font color="#000000">5</font>% <font color="#000000">1</font>.00x ONLINE -
paul@f0:~ % doas zfs list -r zdata/enc
NAME USED AVAIL REFER MOUNTPOINT
zdata/enc 200K 899G 200K /data/enc
<i><font color="silver"># On f1</font></i>
paul@f1:~ % doas zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zdata 928G 956K 928G - - <font color="#000000">0</font>% <font color="#000000">0</font>% <font color="#000000">1</font>.00x ONLINE -
zroot 472G <font color="#000000">11</font>.7G 460G - - <font color="#000000">0</font>% <font color="#000000">2</font>% <font color="#000000">1</font>.00x ONLINE -
paul@f1:~ % doas zfs list -r zdata/enc
NAME USED AVAIL REFER MOUNTPOINT
zdata/enc 200K 899G 200K /data/enc
</pre>
<br />
<h3 style='display: inline' id='configuring-zrepl-with-wireguard-tunnel'>Configuring zrepl with WireGuard tunnel</h3><br />
<br />
<span>Since we have a WireGuard tunnel between f0 and f1, we'll use TCP transport over the secure tunnel instead of SSH. First, check the WireGuard IP addresses:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># Check WireGuard interface IPs</font></i>
paul@f0:~ % ifconfig wg0 | grep inet
inet <font color="#000000">192.168</font>.<font color="#000000">2.130</font> netmask <font color="#000000">0xffffff00</font>
paul@f1:~ % ifconfig wg0 | grep inet
inet <font color="#000000">192.168</font>.<font color="#000000">2.131</font> netmask <font color="#000000">0xffffff00</font>
</pre>
<br />
<h3 style='display: inline' id='configuring-zrepl-on-f0-source'>Configuring zrepl on f0 (source)</h3><br />
<br />
<span>First, create a dedicated dataset for NFS data that will be replicated:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># Create the nfsdata dataset that will hold all data exposed via NFS</font></i>
paul@f0:~ % doas zfs create zdata/enc/nfsdata
</pre>
<br />
<span>Create the zrepl configuration on f0:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<<font color="#808080">'EOF'</font>
global:
logging:
- <b><u><font color="#000000">type</font></u></b>: stdout
level: info
format: human
<b><u><font color="#000000">jobs</font></u></b>:
- name: f0_to_f1
<b><u><font color="#000000">type</font></u></b>: push
connect:
<b><u><font color="#000000">type</font></u></b>: tcp
address: <font color="#808080">"192.168.2.131:8888"</font>
filesystems:
<font color="#808080">"zdata/enc/nfsdata"</font>: <b><u><font color="#000000">true</font></u></b>
<font color="#808080">"zroot/bhyve/fedora"</font>: <b><u><font color="#000000">true</font></u></b>
send:
encrypted: <b><u><font color="#000000">true</font></u></b>
snapshotting:
<b><u><font color="#000000">type</font></u></b>: periodic
prefix: zrepl_
interval: 5m
pruning:
keep_sender:
- <b><u><font color="#000000">type</font></u></b>: last_n
count: <font color="#000000">10</font>
keep_receiver:
- <b><u><font color="#000000">type</font></u></b>: last_n
count: <font color="#000000">10</font>
EOF
</pre>
<br />
<span>Note: We're specifically replicating <span class='inlinecode'>zdata/enc/nfsdata</span> instead of the entire <span class='inlinecode'>zdata/enc</span> dataset. This dedicated dataset will contain all the data we later want to expose via NFS, keeping a clear separation between replicated NFS data and other local encrypted data.</span><br />
<br />
<h3 style='display: inline' id='configuring-zrepl-on-f1-sink'>Configuring zrepl on f1 (sink)</h3><br />
<br />
<span>Create the zrepl configuration on f1:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># First create a dedicated sink dataset</font></i>
paul@f1:~ % doas zfs create zdata/sink
paul@f1:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<<font color="#808080">'EOF'</font>
global:
logging:
- <b><u><font color="#000000">type</font></u></b>: stdout
level: info
format: human
<b><u><font color="#000000">jobs</font></u></b>:
- name: <font color="#808080">"sink"</font>
<b><u><font color="#000000">type</font></u></b>: sink
serve:
<b><u><font color="#000000">type</font></u></b>: tcp
listen: <font color="#808080">"192.168.2.131:8888"</font>
clients:
<font color="#808080">"192.168.2.130"</font>: <font color="#808080">"f0"</font>
recv:
placeholder:
encryption: inherit
root_fs: <font color="#808080">"zdata/sink"</font>
EOF
</pre>
<br />
<h3 style='display: inline' id='enabling-and-starting-zrepl-services'>Enabling and starting zrepl services</h3><br />
<br />
<span>Enable and start zrepl on both hosts:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># On f0</font></i>
paul@f0:~ % doas sysrc zrepl_enable=YES
zrepl_enable: -> YES
paul@f0:~ % doas service zrepl start
Starting zrepl.
<i><font color="silver"># On f1</font></i>
paul@f1:~ % doas sysrc zrepl_enable=YES
zrepl_enable: -> YES
paul@f1:~ % doas service zrepl start
Starting zrepl.
</pre>
<br />
<h3 style='display: inline' id='verifying-replication'>Verifying replication</h3><br />
<br />
<span>Check the replication status:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># On f0, check zrepl status (use raw mode for non-tty)</font></i>
paul@f0:~ % doas zrepl status --mode raw | grep -A<font color="#000000">2</font> <font color="#808080">"Replication"</font>
<font color="#808080">"Replication"</font>:{<font color="#808080">"StartAt"</font>:<font color="#808080">"2025-07-01T22:31:48.712143123+03:00"</font>...
<i><font color="silver"># Check if services are running</font></i>
paul@f0:~ % doas service zrepl status
zrepl is running as pid <font color="#000000">2649</font>.
paul@f1:~ % doas service zrepl status
zrepl is running as pid <font color="#000000">2574</font>.
<i><font color="silver"># Check for zrepl snapshots on source</font></i>
paul@f0:~ % doas zfs list -t snapshot -r zdata/enc | grep zrepl
zdata/enc@zrepl_20250701_193148_000 0B - 176K -
<i><font color="silver"># On f1, verify the replicated datasets </font></i>
paul@f1:~ % doas zfs list -r zdata | grep f0
zdata/f<font color="#000000">0</font> 576K 899G 200K none
zdata/f<font color="#000000">0</font>/zdata 376K 899G 200K none
zdata/f<font color="#000000">0</font>/zdata/enc 176K 899G 176K none
<i><font color="silver"># Check replicated snapshots on f1</font></i>
paul@f1:~ % doas zfs list -t snapshot -r zdata | grep zrepl
zdata/f<font color="#000000">0</font>/zdata/enc@zrepl_20250701_193148_000 0B - 176K -
zdata/f<font color="#000000">0</font>/zdata/enc@zrepl_20250701_194148_000 0B - 176K -
</pre>
<br />
<h3 style='display: inline' id='monitoring-replication'>Monitoring replication</h3><br />
<br />
<span>You can monitor the replication progress with:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># Real-time status</font></i>
paul@f0:~ % doas zrepl status --mode interactive
<i><font color="silver"># Check specific job details</font></i>
paul@f0:~ % doas zrepl status --job f0_to_f1
</pre>
<br />
<span>With this setup, both <span class='inlinecode'>zdata/enc/nfsdata</span> and <span class='inlinecode'>zroot/bhyve/fedora</span> on f0 will be automatically replicated to f1 every 5 minutes, with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep the last 10 snapshots while managing disk space efficiently.</span><br />
<br />
<span>The replicated data appears on f1 under <span class='inlinecode'>zdata/sink/</span> with the source host and dataset hierarchy preserved:</span><br />
<br />
<ul>
<li><span class='inlinecode'>zdata/enc/nfsdata</span> → <span class='inlinecode'>zdata/sink/f0/zdata/enc/nfsdata</span></li>
<li><span class='inlinecode'>zroot/bhyve/fedora</span> → <span class='inlinecode'>zdata/sink/f0/zroot/bhyve/fedora</span></li>
</ul><br />
<span>This is by design - zrepl preserves the complete path from the source to ensure there are no conflicts when replicating from multiple sources. The replication uses the WireGuard tunnel for secure, encrypted transport between nodes.</span><br />
<br />
<h3 style='display: inline' id='a-note-about-the-bhyve-vm-replication'>A note about the Bhyve VM replication</h3><br />
<br />
<span>While replicating a Bhyve VM (Fedora in this case) is slightly off-topic for the f3s series, I've included it here as it demonstrates zrepl's flexibility. This is a development VM I use occasionally to log in remotely for certain development tasks. Having it replicated ensures I have a backup copy available on f1 if needed.</span><br />
<br />
<h3 style='display: inline' id='quick-status-check-commands'>Quick status check commands</h3><br />
<br />
<span>Here are the essential commands to monitor replication status:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># On the source node (f0) - check if replication is active</font></i>
paul@f0:~ % doas zrepl status --job f0_to_f1 | grep -E <font color="#808080">'(State|Last)'</font>
State: <b><u><font color="#000000">done</font></u></b>
LastError:
<i><font color="silver"># List all zrepl snapshots on source</font></i>
paul@f0:~ % doas zfs list -t snapshot | grep zrepl
zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 200K -
zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - <font color="#000000">2</font>.97G -
<i><font color="silver"># On the sink node (f1) - verify received datasets</font></i>
paul@f1:~ % doas zfs list -r zdata/sink
NAME USED AVAIL REFER MOUNTPOINT
zdata/sink <font color="#000000">3</font>.0G 896G 200K /data/sink
zdata/sink/f<font color="#000000">0</font> <font color="#000000">3</font>.0G 896G 200K none
zdata/sink/f<font color="#000000">0</font>/zdata 472K 896G 200K none
zdata/sink/f<font color="#000000">0</font>/zdata/enc 272K 896G 200K none
zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata 176K 896G 176K none
zdata/sink/f<font color="#000000">0</font>/zroot <font color="#000000">2</font>.9G 896G 200K none
zdata/sink/f<font color="#000000">0</font>/zroot/bhyve <font color="#000000">2</font>.9G 896G 200K none
zdata/sink/f<font color="#000000">0</font>/zroot/bhyve/fedora <font color="#000000">2</font>.9G 896G <font color="#000000">2</font>.97G none
<i><font color="silver"># Check received snapshots on sink</font></i>
paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep zrepl | wc -l
<font color="#000000">3</font>
<i><font color="silver"># Monitor replication progress in real-time (on source)</font></i>
paul@f0:~ % doas zrepl status --mode interactive
<i><font color="silver"># Check last replication time (on source)</font></i>
paul@f0:~ % doas zrepl status --job f0_to_f1 | grep -A<font color="#000000">1</font> <font color="#808080">"Replication"</font>
Replication:
Status: Idle (last run: <font color="#000000">2025</font>-<font color="#000000">07</font>-01T22:<font color="#000000">41</font>:<font color="#000000">48</font>)
<i><font color="silver"># View zrepl logs for troubleshooting</font></i>
paul@f0:~ % doas tail -<font color="#000000">20</font> /var/log/zrepl.log | grep -E <font color="#808080">'(error|warn|replication)'</font>
</pre>
<br />
<span>These commands provide a quick way to verify that:</span><br />
<br />
<ul>
<li>Replication jobs are running without errors</li>
<li>Snapshots are being created on the source</li>
<li>Data is being received on the sink</li>
<li>The replication schedule is being followed</li>
</ul><br />
<h3 style='display: inline' id='verifying-replication-after-reboot'>Verifying replication after reboot</h3><br />
<br />
<span>The zrepl service is configured to start automatically at boot. After rebooting both hosts:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f0:~ % uptime
<font color="#000000">11</font>:17PM up <font color="#000000">1</font> min, <font color="#000000">0</font> users, load averages: <font color="#000000">0.16</font>, <font color="#000000">0.06</font>, <font color="#000000">0.02</font>
paul@f0:~ % doas service zrepl status
zrepl is running as pid <font color="#000000">2366</font>.
paul@f1:~ % doas service zrepl status
zrepl is running as pid <font color="#000000">2309</font>.
<i><font color="silver"># Check that new snapshots are being created and replicated</font></i>
paul@f0:~ % doas zfs list -t snapshot | grep zrepl | tail -<font color="#000000">2</font>
zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 200K -
zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - <font color="#000000">2</font>.97G -
paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep <font color="#000000">202530</font>
zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 176K -
zdata/sink/f<font color="#000000">0</font>/zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - <font color="#000000">2</font>.97G -
</pre>
<br />
<span>The timestamps confirm that replication resumed automatically after the reboot, ensuring continuous data protection.</span><br />
<br />
<h3 style='display: inline' id='important-note-about-failover-limitations'>Important note about failover limitations</h3><br />
<br />
<span>The current zrepl setup provides **backup/disaster recovery** but not automatic failover. The replicated datasets on f1 are not mounted by default (<span class='inlinecode'>mountpoint=none</span>). In case f0 fails:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># Manual steps needed on f1 to activate the replicated data:</font></i>
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/data/nfsdata zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
paul@f1:~ % doas zfs mount zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
</pre>
<br />
<span>However, this creates a **split-brain problem**: when f0 comes back online, both systems would have diverged data. Resolving this requires careful manual intervention to:</span><br />
<br />
<span>1. Stop the original replication</span><br />
<span>2. Sync changes from f1 back to f0</span><br />
<span>3. Re-establish normal replication</span><br />
<br />
<span>For true high-availability NFS, you might consider:</span><br />
<br />
<ul>
<li>**Shared storage** (like iSCSI) with proper clustering</li>
<li>**GlusterFS** or similar distributed filesystems</li>
<li>**Manual failover with ZFS replication** (as we have here)</li>
</ul><br />
<span>Note: While HAST+CARP is often suggested for HA storage, it can cause filesystem corruption in practice, especially with ZFS. The block-level replication of HAST doesn't understand ZFS's transactional model, leading to inconsistent states during failover. </span><br />
<br />
<span>The current zrepl setup, despite requiring manual intervention, is actually safer because:</span><br />
<br />
<ul>
<li>ZFS snapshots are always consistent</li>
<li>Replication is ZFS-aware (not just block-level)</li>
<li>You have full control over the failover process</li>
<li>No risk of split-brain corruption</li>
</ul><br />
<h3 style='display: inline' id='mounting-the-nfs-datasets'>Mounting the NFS datasets</h3><br />
<br />
<span>To make the nfsdata accessible on both nodes, we need to mount them. On f0, this is straightforward:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># On f0 - set mountpoint for the primary nfsdata</font></i>
paul@f0:~ % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/data/nfs zdata/enc/nfsdata
paul@f0:~ % doas mkdir -p /data/nfs
<i><font color="silver"># Verify it's mounted</font></i>
paul@f0:~ % df -h /data/nfs
Filesystem Size Used Avail Capacity Mounted on
zdata/enc/nfsdata 899G 204K 899G <font color="#000000">0</font>% /data/nfs
</pre>
<br />
<span>On f1, we need to handle the encryption key and mount the standby copy:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># On f1 - first check encryption status</font></i>
paul@f1:~ % doas zfs get keystatus zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
NAME PROPERTY VALUE SOURCE
zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata keystatus unavailable -
<i><font color="silver"># Load the encryption key (using f0's key stored on the USB)</font></i>
paul@f1:~ % doas zfs load-key -L file:///keys/f<font color="#000000">0</font>.lan.buetow.org:zdata.key \
zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
<i><font color="silver"># Set mountpoint and mount (same path as f0 for easier failover)</font></i>
paul@f1:~ % doas mkdir -p /data/nfs
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/data/nfs zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
paul@f1:~ % doas zfs mount zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
<i><font color="silver"># Make it read-only to prevent accidental writes that would break replication</font></i>
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> <b><u><font color="#000000">readonly</font></u></b>=on zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
<i><font color="silver"># Verify</font></i>
paul@f1:~ % df -h /data/nfs
Filesystem Size Used Avail Capacity Mounted on
zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata 896G 204K 896G <font color="#000000">0</font>% /data/nfs
</pre>
<br />
<span>Note: The dataset is mounted at the same path (<span class='inlinecode'>/data/nfs</span>) on both hosts to simplify failover procedures. The dataset on f1 is set to <span class='inlinecode'>readonly=on</span> to prevent accidental modifications that would break replication.</span><br />
<br />
<span>**CRITICAL WARNING**: Do NOT write to <span class='inlinecode'>/data/nfs/</span> on f1! Any modifications will break the replication. If you accidentally write to it, you'll see this error:</span><br />
<br />
<pre>
cannot receive incremental stream: destination zdata/sink/f0/zdata/enc/nfsdata has been modified
since most recent snapshot
</pre>
<br />
<span>To fix a broken replication after accidental writes:</span><br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># Option 1: Rollback to the last common snapshot (loses local changes)</font></i>
paul@f1:~ % doas zfs rollback zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@zrepl_20250701_204054_000
<i><font color="silver"># Option 2: Make it read-only to prevent accidents</font></i>
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> <b><u><font color="#000000">readonly</font></u></b>=on zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
</pre>
<br />
<span>To ensure the encryption key is loaded automatically after reboot on f1:</span><br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>paul@f1:~ % doas sysrc zfskeys_datasets=<font color="#808080">"zdata/sink/f0/zdata/enc/nfsdata"</font>
</pre>
<br />
<h3 style='display: inline' id='failback-scenario-syncing-changes-from-f1-back-to-f0'>Failback scenario: Syncing changes from f1 back to f0</h3><br />
<br />
<span>In a disaster recovery scenario where f0 has failed and f1 has taken over, you'll need to sync changes back when f0 returns. Here's how to failback:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># On f1: First, make the dataset writable (if it was readonly)</font></i>
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> <b><u><font color="#000000">readonly</font></u></b>=off zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
<i><font color="silver"># Create a snapshot of the current state</font></i>
paul@f1:~ % doas zfs snapshot zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@failback
<i><font color="silver"># On f0: Stop any services using the dataset</font></i>
paul@f0:~ % doas service nfsd stop <i><font color="silver"># If NFS is running</font></i>
<i><font color="silver"># Send the snapshot from f1 to f0, forcing a rollback</font></i>
<i><font color="silver"># This WILL DESTROY any data on f0 that's not on f1!</font></i>
paul@f1:~ % doas zfs send -R zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@failback | \
ssh f0 <font color="#808080">"doas zfs recv -F zdata/enc/nfsdata"</font>
<i><font color="silver"># Alternative: If you want to see what would be received first</font></i>
paul@f1:~ % doas zfs send -R zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@failback | \
ssh f0 <font color="#808080">"doas zfs recv -nv -F zdata/enc/nfsdata"</font>
<i><font color="silver"># After successful sync, on f0:</font></i>
paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@failback
<i><font color="silver"># On f1: Make it readonly again and destroy the failback snapshot</font></i>
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> <b><u><font color="#000000">readonly</font></u></b>=on zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
paul@f1:~ % doas zfs destroy zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@failback
<i><font color="silver"># Stop zrepl services first - CRITICAL!</font></i>
paul@f0:~ % doas service zrepl stop
paul@f1:~ % doas service zrepl stop
<i><font color="silver"># Clean up any zrepl snapshots on f0</font></i>
paul@f0:~ % doas zfs list -t snapshot -r zdata/enc/nfsdata | grep zrepl | \
awk <font color="#808080">'{print $1}'</font> | xargs -I {} doas zfs destroy {}
<i><font color="silver"># Clean up and destroy the entire replicated structure on f1</font></i>
<i><font color="silver"># First release any holds</font></i>
paul@f1:~ % doas zfs holds -r zdata/sink/f<font color="#000000">0</font> | grep -v NAME | \
awk <font color="#808080">'{print $2, $1}'</font> | <b><u><font color="#000000">while</font></u></b> <b><u><font color="#000000">read</font></u></b> tag snap; <b><u><font color="#000000">do</font></u></b>
doas zfs release <font color="#808080">"$tag"</font> <font color="#808080">"$snap"</font>
<b><u><font color="#000000">done</font></u></b>
<i><font color="silver"># Then destroy the entire f0 tree</font></i>
paul@f1:~ % doas zfs destroy -rf zdata/sink/f<font color="#000000">0</font>
<i><font color="silver"># Create parent dataset structure on f1</font></i>
paul@f1:~ % doas zfs create -p zdata/sink/f<font color="#000000">0</font>/zdata/enc
<i><font color="silver"># Create a fresh manual snapshot to establish baseline</font></i>
paul@f0:~ % doas zfs snapshot zdata/enc/nfsdata@manual_baseline
<i><font color="silver"># Send this snapshot to f1</font></i>
paul@f0:~ % doas zfs send -w zdata/enc/nfsdata@manual_baseline | \
ssh f1 <font color="#808080">"doas zfs recv zdata/sink/f0/zdata/enc/nfsdata"</font>
<i><font color="silver"># Clean up the manual snapshot</font></i>
paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@manual_baseline
paul@f1:~ % doas zfs destroy zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@manual_baseline
<i><font color="silver"># Set mountpoint and make readonly on f1</font></i>
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/data/nfs zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> <b><u><font color="#000000">readonly</font></u></b>=on zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
<i><font color="silver"># Load encryption key and mount on f1</font></i>
paul@f1:~ % doas zfs load-key -L file:///keys/f<font color="#000000">0</font>.lan.buetow.org:zdata.key \
zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
paul@f1:~ % doas zfs mount zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
<i><font color="silver"># Now restart zrepl services</font></i>
paul@f0:~ % doas service zrepl start
paul@f1:~ % doas service zrepl start
<i><font color="silver"># Verify replication is working</font></i>
paul@f0:~ % doas zrepl status --job f0_to_f1
</pre>
<br />
<span>**Important notes about failback**:</span><br />
<br />
<ul>
<li>The <span class='inlinecode'>-F</span> flag forces a rollback on f0, destroying any local changes</li>
<li>Replication often won't resume automatically after a forced receive</li>
<li>You must clean up old zrepl snapshots on both sides</li>
<li>Creating a manual snapshot helps re-establish the replication relationship</li>
<li>Always verify replication status after the failback procedure</li>
<li>The first replication after failback will be a full send of the current state</li>
</ul><br />
<h3 style='display: inline' id='testing-the-failback-scenario'>Testing the failback scenario</h3><br />
<br />
<span>Here's a real test of the failback procedure:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="silver"># Simulate failure: Stop replication on f0</font></i>
paul@f0:~ % doas service zrepl stop
<i><font color="silver"># On f1: Take over by making the dataset writable</font></i>
paul@f1:~ % doas zfs <b><u><font color="#000000">set</font></u></b> <b><u><font color="#000000">readonly</font></u></b>=off zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata
<i><font color="silver"># Write some data on f1 during the "outage"</font></i>
paul@f1:~ % echo <font color="#808080">'Data written on f1 during failover'</font> | doas tee /data/nfs/failover-data.txt
Data written on f1 during failover
<i><font color="silver"># Now perform failback when f0 comes back online</font></i>
<i><font color="silver"># Create snapshot on f1</font></i>
paul@f1:~ % doas zfs snapshot zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@failback
<i><font color="silver"># Send data back to f0 (note: we had to send to a temporary dataset due to holds)</font></i>
paul@f1:~ % doas zfs send -Rw zdata/sink/f<font color="#000000">0</font>/zdata/enc/nfsdata@failback | \
ssh f0 <font color="#808080">"doas zfs recv -F zdata/enc/nfsdata_temp"</font>
<i><font color="silver"># On f0: Rename datasets to complete failback</font></i>
paul@f0:~ % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=none zdata/enc/nfsdata
paul@f0:~ % doas zfs rename zdata/enc/nfsdata zdata/enc/nfsdata_old
paul@f0:~ % doas zfs rename zdata/enc/nfsdata_temp zdata/enc/nfsdata
<i><font color="silver"># Load encryption key and mount</font></i>
paul@f0:~ % doas zfs load-key -L file:///keys/f<font color="#000000">0</font>.lan.buetow.org:zdata.key zdata/enc/nfsdata
paul@f0:~ % doas zfs mount zdata/enc/nfsdata
<i><font color="silver"># Verify the data from f1 is now on f0</font></i>
paul@f0:~ % ls -la /data/nfs/
total <font color="#000000">18</font>
drwxr-xr-x <font color="#000000">2</font> root wheel <font color="#000000">4</font> Jul <font color="#000000">2</font> <font color="#000000">00</font>:<font color="#000000">01</font> .
drwxr-xr-x <font color="#000000">4</font> root wheel <font color="#000000">4</font> Jul <font color="#000000">1</font> <font color="#000000">23</font>:<font color="#000000">41</font> ..
-rw-r--r-- <font color="#000000">1</font> root wheel <font color="#000000">35</font> Jul <font color="#000000">2</font> <font color="#000000">00</font>:<font color="#000000">01</font> failover-data.txt
-rw-r--r-- <font color="#000000">1</font> root wheel <font color="#000000">12</font> Jul <font color="#000000">1</font> <font color="#000000">23</font>:<font color="#000000">34</font> hello.txt
</pre>
<br />
<span>Success! The failover data from f1 is now on f0. To resume normal replication, you would need to:</span><br />
<br />
<span>1. Clean up old snapshots on both sides</span><br />
<span>2. Create a new manual baseline snapshot</span><br />
<span>3. Restart zrepl services</span><br />
<br />
<span>**Key learnings from the test**:</span><br />
<br />
<ul>
<li>The <span class='inlinecode'>-w</span> flag is essential for encrypted datasets</li>
<li>Dataset holds can complicate the process (consider sending to a temporary dataset)</li>
<li>The encryption key must be loaded after receiving the dataset</li>
<li>Always verify data integrity before resuming normal operations</li>
</ul><br />
<span>ZFS auto scrubbing....~?</span><br />
<br />
<span>Backup of the keys on the key locations (all keys on all 3 USB keys)</span><br />
<br />
<span>Other *BSD-related posts:</span><br />
<br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span></span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
<br />
<span>https://forums.freebsd.org/threads/hast-and-zfs-with-carp-failover.29639/</span><br />
<br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span></span><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
<p class="footer">
Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
</p>
</body>
</html>
|
