1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
|
# f3s: Kubernetes with FreeBSD - Part 6: Storage
> Published at 2025-04-04T23:21:01+03:00
This is the sixth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.
<< template::inline::index f3s-kubernetes-with-freebsd-part
=> ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
<< template::inline::toc
## Introduction
In this blog post, we are going to extend the Beelinks with some additional storage.
Some photos here, describe why there are 2 different models of SSD drives (replication etc)
```
paul@f0:/ % doas camcontrol devlist
<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0)
<Samsung SSD 870 EVO 1TB SVT03B6Q> at scbus1 target 0 lun 0 (pass1,ada1)
<Generic Flash Disk 8.07> at scbus2 target 0 lun 0 (da0,pass2)
paul@f0:/ %
```
```
paul@f1:/ % doas camcontrol devlist
<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0)
<CT1000BX500SSD1 M6CR072> at scbus1 target 0 lun 0 (pass1,ada1)
<Generic Flash Disk 8.07> at scbus2 target 0 lun 0 (da0,pass2)
paul@f1:/ %
```
## UFS Setup
```sh
paul@f0:/ % doas newfs /dev/da0
/dev/da0: 15000.0MB (30720000 sectors) block size 32768, fragment size 4096
using 24 cylinder groups of 625.22MB, 20007 blks, 80128 inodes.
with soft updates
super-block backups (for fsck_ffs -b #) at:
192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880, 8963328, 10243776,
11524224, 12804672, 14085120, 15365568, 16646016, 17926464, 19206912,k 20487360,
...
paul@f0:/ % echo '/dev/da0 /keys ufs rw 0 2' | doas tee -a /etc/fstab
/dev/da0 /keys ufs rw 0 2
paul@f0:/ % doas mkdir /keys
paul@f0:/ % doas mount /keys
paul@f0:/ % df | grep keys
/dev/da0 14877596 8 13687384 0% /keys
```
## ZFS Setup
```sh
paul@f0:/dev % doas zpool create -m /data zdata /dev/ada1
paul@f0:/dev % zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zdata 928G 432K 928G - - 0% 0% 1.00x ONLINE -
zroot 472G 19.8G 452G - - 0% 4% 1.00x ONLINE -
```
### Encryption
USB key for key location
```sh
paul@f0:/keys % doas vm stop rocky
Sending ACPI shutdown to rocky
paul@f0:/keys % doas vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
rocky default uefi 4 14G - Yes [1] Stopped
paul@f0:/keys % doas zfs rename zroot/bhyve zroot/bhyve_old
paul@f0:/keys % doas zfs set mountpoint=/mnt zroot/bhyve_old
paul@f0:/keys % doas zfs snapshot zroot/bhyve_old/rocky@hamburger
paul@f0:/keys % doas openssl rand -out /keys/`hostname`:bhyve.key 32
paul@f0:/keys % doas openssl rand -out /keys/`hostname`:zdata.key 32
paul@f0:/keys % ls -ltr
total 8
-rw-r--r-- 1 root wheel 16 May 25 11:54 f0.lan.buetow.org:bhyve.key
-rw-r--r-- 1 root wheel 16 May 25 11:54 f0.lan.buetow.org:zdata.key
paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:bhyve.key zroot/bhyve
paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:zdata.key zdata/enc
paul@f0:/keys % doas zfs set mountpoint=/zroot/bhyve zroot/bhyve
paul@f0:/keys % doas zfs set mountpoint=/zroot/bhyve/rocky zroot/bhyve/rocky
paul@f0:/keys % doas zfs send zroot/bhyve_old/rocky@hamburger | doas zfs recv zroot/bhyve/rocky
paul@f0:/keys % doas cp -Rp /mnt/.config /zroot/bhyve/
paul@f0:/keys % doas cp -Rp /mnt/.img /zroot/bhyve/
paul@f0:/keys % doas cp -Rp /mnt/.templates /zroot/bhyve/
paul@f0:/keys % doas cp -Rp /mnt/.iso /zroot/bhyve/
paul@f0:/keys % doas sysrc zfskeys_enable=YES
zfskeys_enable: -> YES
```
Copied over all the keys from the partner node to each node, so they backup each other:
```sh
paul@f0:/keys % doas chown root *
paul@f0:/keys % doas chmod 400 *
paul@f0:/keys % ls -ltr
total 24
-r-------- 1 root paul 16 May 25 11:56 f0.lan.buetow.org:zdata.key
-r-------- 1 root paul 16 May 25 11:56 f0.lan.buetow.org:bhyve.key
-r-------- 1 root paul 16 May 25 11:56 f1.lan.buetow.org:zdata.key
-r-------- 1 root paul 16 May 25 11:56 f1.lan.buetow.org:bhyve.key
-r-------- 1 root paul 16 May 25 11:57 f2.lan.buetow.org:zdata.key
-r-------- 1 root paul 16 May 25 11:57 f2.lan.buetow.org:bhyve.key
```
```sh
paul@f0:/keys % doas vm init
paul@f0:/keys % doas reboot
.
.
.
paul@f0:~ % doas vm list
paul@f0:~ % doas vm list
NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
rocky default uefi 4 14G 0.0.0.0:5900 Yes [1] Running (2265)
```
```sh
paul@f0:~ % doas zfs destroy -R zroot/bhyve_old
paul@f0:~ % zfs get all zdata/enc | grep -E '(encryption|key)'
zdata/enc encryption aes-256-gcm -
zdata/enc keylocation file:///keys/f0.lan.buetow.org:zdata.key local
zdata/enc keyformat raw -
zdata/enc encryptionroot zdata/enc -
zdata/enc keystatus available -
paul@f0:~ % zfs get all zroot/bhyve | grep -E '(encryption|key)'
zroot/bhyve encryption aes-256-gcm -
zroot/bhyve keylocation file:///keys/f0.lan.buetow.org:bhyve.key local
zroot/bhyve keyformat raw -
zroot/bhyve encryptionroot zroot/bhyve -
zroot/bhyve keystatus available -
paul@f0:~ % zfs get all zroot/bhyve/rocky | grep -E '(encryption|key)'
zroot/bhyve/rocky encryption aes-256-gcm -
zroot/bhyve/rocky keylocation none default
zroot/bhyve/rocky keyformat raw -
zroot/bhyve/rocky encryptionroot zroot/bhyve -
zroot/bhyve/rocky keystatus available -
```
```
paul@f0:~ % zpool status
pool: zdata
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zdata ONLINE 0 0 0
ada1 ONLINE 0 0 0
errors: No known data errors
pool: zroot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
ada0p4 ONLINE 0 0 0
errors: No known data errors
```
ZFS auto scrubbing....~?
Backup of the keys on the key locations (all keys on all 3 USB keys)
Other *BSD-related posts:
<< template::inline::rindex bsd
E-Mail your comments to `paul@nospam.buetow.org`
=> ../ Back to the main site
|
