1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>KISS high-availability with OpenBSD</title>
<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
<link rel="stylesheet" href="../style.css" />
<link rel="stylesheet" href="style-override.css" />
</head>
<body>
<div class="rfx-overlay-grid"></div>
<div class="rfx-overlay-scanlines"></div>
<div id="rfx-stars"></div>
<div class="rfx-vignette"></div>
<p class="header">
<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi">Gemini</a>
</p>
<h1 style='display: inline' id='kiss-high-availability-with-openbsd'>KISS high-availability with OpenBSD</h1><br />
<br />
<span class='quote'>Published at 2024-03-30T22:12:56+02:00</span><br />
<br />
<span>I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like (in unsorted and slightly unrelated order) BGP, LVS/IPVS, ldirectord, Pacemaker, STONITH, scripted VIP failover via ARP, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work. </span><br />
<br />
<span>But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space - something as KISS (keep it simple and stupid) as possible.</span><br />
<br />
<span>It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway.</span><br />
<br />
<span class='quote'>PS: ASCII-art below reflects an OpenBSD under-water world with all the tools available in the base system.</span><br />
<br />
<pre>
Art by Michael J. Penick (mod. by Paul B.)
ACME-sky
__________
/ nsd tower\ (
/____________\ (\) awk-ward
|:_:_:_:_:_| )) plant
|_:_,--.:_:| dig-bubble (\// )
|:_:|__|_:_| relayd-castle _ ) )) ((
_ |_ _ :_:| _ _ _ (_) (((( /)\`
| |_| |_| | _| | |_| |_| | o \\)) (( (
\_:_:_:_:/|_|_|_|\:_:_:_:_/ . (( ))))
|_,-._:_:_:_:_:_:_:_.-,_| )) ((//
|:|_|:_:_:,---,:_:_:|_|:| ,-. )/
|_:_:_:_,'puffy `,_:_:_:_| _ o ,;'))((
|:_:_:_/ _ | _ \_:_:_:| (_O (( ))
_____|_:_:_| (o)-(o) |_:_:_|--'`-. ,--. ksh under-water (((\'/
', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\ /,---.~ goat \`))
. ` |_:_:_| \`-'/ |_:_:_|. ` . ` /()\.__( ) .,-----'`-\(( sed-root
', ;|:_:_:| `-' |:_:_:| ', ; ', ; `--'| \ ', ; ', ; ',')).,--
. ` MJP ` . ` . ` . ` . httpd-soil ` . . ` . ` . ` . ` . `
', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ;
</pre>
<br />
<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
<br />
<ul>
<li><a href='#kiss-high-availability-with-openbsd'>KISS high-availability with OpenBSD</a></li>
<li>⇢ <a href='#my-auto-failover-requirements'>My auto-failover requirements</a></li>
<li>⇢ <a href='#my-ha-solution'>My HA solution</a></li>
<li>⇢ ⇢ <a href='#only-openbsd-base-installation-required'>Only OpenBSD base installation required</a></li>
<li>⇢ ⇢ <a href='#fairly-cheap-and-geo-redundant'>Fairly cheap and geo-redundant</a></li>
<li>⇢ ⇢ <a href='#failover-time-and-split-brain'>Failover time and split-brain</a></li>
<li>⇢ ⇢ <a href='#failover-support-for-multiple-protocols'>Failover support for multiple protocols</a></li>
<li>⇢ ⇢ <a href='#let-s-encrypt-tls-certificates'>Let's encrypt TLS certificates</a></li>
<li>⇢ ⇢ <a href='#monitoring'>Monitoring</a></li>
<li>⇢ ⇢ <a href='#rex-automation'>Rex automation</a></li>
<li>⇢ <a href='#more-ha'>More HA</a></li>
</ul><br />
<h2 style='display: inline' id='my-auto-failover-requirements'>My auto-failover requirements</h2><br />
<br />
<ul>
<li>Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible. </li>
<li>Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month already!).</li>
<li>It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.</li>
<li>It should be geo-redundant. </li>
<li>It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.</li>
<li>Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.</li>
<li>Let's Encrypt TLS certificates should always work (before and after a failover).</li>
<li>Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover.</li>
<li>Don't configure everything manually. The configuration should be automated and reproducible.</li>
</ul><br />
<h2 style='display: inline' id='my-ha-solution'>My HA solution</h2><br />
<br />
<h3 style='display: inline' id='only-openbsd-base-installation-required'>Only OpenBSD base installation required</h3><br />
<br />
<span>My HA solution for Web and Gemini is based on DNS (OpenBSD's <span class='inlinecode'>nsd</span>) and a simple shell script (OpenBSD's <span class='inlinecode'>ksh</span> and some little <span class='inlinecode'>sed</span> and <span class='inlinecode'>awk</span> and <span class='inlinecode'>grep</span>). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system.</span><br />
<br />
<a class='textlink' href='https://man.OpenBSD.org/nsd.8'>https://man.OpenBSD.org/nsd.8</a><br />
<a class='textlink' href='https://man.OpenBSD.org/ksh'>https://man.OpenBSD.org/ksh</a><br />
<a class='textlink' href='https://man.OpenBSD.org/awk'>https://man.OpenBSD.org/awk</a><br />
<a class='textlink' href='https://man.OpenBSD.org/sed'>https://man.OpenBSD.org/sed</a><br />
<a class='textlink' href='https://man.OpenBSD.org/dig'>https://man.OpenBSD.org/dig</a><br />
<a class='textlink' href='https://man.OpenBSD.org/ftp'>https://man.OpenBSD.org/ftp</a><br />
<a class='textlink' href='https://man.OpenBSD.org/cron'>https://man.OpenBSD.org/cron</a><br />
<br />
<span>I also used the <span class='inlinecode'>dig</span> (for DNS checks) and <span class='inlinecode'>ftp</span> (for HTTP/HTTPS checks) programs. </span><br />
<br />
<span>The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The <span class='inlinecode'>ksh</span> script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM: </span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#!/bin/ksh</font></i>
<font color="#ff0000">ZONES_DIR</font><font color="#F3E651">=</font><font color="#ff0000">/var/nsd/zones/master</font><font color="#F3E651">/</font>
<font color="#ff0000">DEFAULT_MASTER</font><font color="#F3E651">=</font><font color="#ff0000">fishfinger</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#ff0000">DEFAULT_STANDBY</font><font color="#F3E651">=</font><font color="#ff0000">blowfish</font><font color="#F3E651">.</font><font color="#ff0000">buetow</font><font color="#F3E651">.</font><font color="#ff0000">org</font>
<font color="#7bc710">determine_master_and_standby ()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">master</font><font color="#F3E651">=</font><font color="#ff0000">$DEFAULT_MASTER</font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">standby</font><font color="#F3E651">=</font><font color="#ff0000">$DEFAULT_STANDBY</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> -i </font><font color="#ff0000">health_ok</font><font color="#F3E651">=</font><font color="#bb00ff">1</font>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000"> ftp -</font><font color="#bb00ff">4</font><font color="#ff0000"> -o - https</font><font color="#F3E651">://</font><font color="#ff0000">$master</font><font color="#ff0000">/index</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">|</font><font color="#ff0000"> grep -q </font><font color="#bb00ff">"Welcome to $master"</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"https://$master/index.txt IPv4 health check failed"</font>
<font color="#ff0000"> </font><font color="#ff0000">health_ok</font><font color="#F3E651">=</font><font color="#bb00ff">0</font>
<font color="#ff0000"> </font><b><font color="#ffffff">elif</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000"> ftp -</font><font color="#bb00ff">6</font><font color="#ff0000"> -o - https</font><font color="#F3E651">://</font><font color="#ff0000">$master</font><font color="#ff0000">/index</font><font color="#F3E651">.</font><font color="#ff0000">txt </font><font color="#F3E651">|</font><font color="#ff0000"> grep -q </font><font color="#bb00ff">"Welcome to $master"</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"https://$master/index.txt IPv6 health check failed"</font>
<font color="#ff0000"> </font><font color="#ff0000">health_ok</font><font color="#F3E651">=</font><font color="#bb00ff">0</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#ff0000">$health_ok</font><font color="#ff0000"> -eq </font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">tmp</font><font color="#F3E651">=</font><font color="#ff0000">$master</font>
<font color="#ff0000"> </font><font color="#ff0000">master</font><font color="#F3E651">=</font><font color="#ff0000">$standby</font>
<font color="#ff0000"> </font><font color="#ff0000">standby</font><font color="#F3E651">=</font><font color="#ff0000">$tmp</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000"> </font><font color="#F3E651">.</font>
<font color="#ff0000">}</font>
</pre>
<br />
<span>The failover scripts looks for the <span class='inlinecode'> ; Enable failover</span> string in the DNS zone files and swaps the <span class='inlinecode'>A</span> and <span class='inlinecode'>AAAA</span> records of the DNS entries accordingly:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#ff0000">fishfinger$ grep failover /var/nsd/zones/master/foo</font><font color="#F3E651">.</font><font color="#ff0000">zone</font><font color="#F3E651">.</font><font color="#ff0000">zone</font>
<font color="#ff0000"> </font><font color="#bb00ff">300</font><font color="#ff0000"> IN A </font><font color="#bb00ff">46.23</font><font color="#F3E651">.</font><font color="#bb00ff">94.99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000"> </font><font color="#bb00ff">300</font><font color="#ff0000"> IN AAAA 2a03</font><font color="#F3E651">:</font><font color="#bb00ff">6000</font><font color="#F3E651">:</font><font color="#ff0000">6f67</font><font color="#F3E651">:</font><font color="#bb00ff">624</font><font color="#F3E651">::</font><font color="#bb00ff">99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">www </font><font color="#bb00ff">300</font><font color="#ff0000"> IN A </font><font color="#bb00ff">46.23</font><font color="#F3E651">.</font><font color="#bb00ff">94.99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">www </font><font color="#bb00ff">300</font><font color="#ff0000"> IN AAAA 2a03</font><font color="#F3E651">:</font><font color="#bb00ff">6000</font><font color="#F3E651">:</font><font color="#ff0000">6f67</font><font color="#F3E651">:</font><font color="#bb00ff">624</font><font color="#F3E651">::</font><font color="#bb00ff">99</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">standby </font><font color="#bb00ff">300</font><font color="#ff0000"> IN A </font><font color="#bb00ff">23.88</font><font color="#F3E651">.</font><font color="#bb00ff">35.144</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
<font color="#ff0000">standby </font><font color="#bb00ff">300</font><font color="#ff0000"> IN AAAA 2a01</font><font color="#F3E651">:</font><font color="#ff0000">4f8</font><font color="#F3E651">:</font><font color="#ff0000">c17</font><font color="#F3E651">:</font><font color="#ff0000">20f1</font><font color="#F3E651">::</font><font color="#bb00ff">42</font><font color="#ff0000"> </font><font color="#F3E651">;</font><font color="#ff0000"> Enable failover</font>
</pre>
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><font color="#7bc710">transform ()</font><font color="#ff0000"> {</font>
<font color="#ff0000"> sed -E </font><font color="#bb00ff">'</font>
<font color="#bb00ff"> /IN A .*; Enable failover/ {</font>
<font color="#bb00ff"> /^standby/! {</font>
<font color="#bb00ff"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN A '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/master_a</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> /^standby/ {</font>
<font color="#bb00ff"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN A '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/standby_a</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> /IN AAAA .*; Enable failover/ {</font>
<font color="#bb00ff"> /^standby/! {</font>
<font color="#bb00ff"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN AAAA '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/master_aaaa</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> /^standby/ {</font>
<font color="#bb00ff"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> 300 IN AAAA '</font><font color="#ff0000">$(</font><font color="#ff0000">cat /var/nsd/run/standby_aaaa</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> / ; serial/ {</font>
<font color="#bb00ff"> s/^( +) ([0-9]+) .*; (.*)/</font><font color="#ffffff">\1</font><font color="#bb00ff"> '</font><font color="#ff0000">$(</font><font color="#ff0000">date </font><font color="#F3E651">+%</font><font color="#ff0000">s</font><font color="#F3E651">)</font><font color="#bb00ff">' ; </font><font color="#ffffff">\3</font><font color="#bb00ff">/;</font>
<font color="#bb00ff"> }</font>
<font color="#bb00ff"> '</font>
<font color="#ff0000">}</font>
</pre>
<br />
<span>After the failover, the script reloads <span class='inlinecode'>nsd</span> and performs a sanity check to see if DNS still works. If not, a rollback will be performed:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab">#! Race condition !#</font></i>
<font color="#ff0000"> </font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> -f </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> mv </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak </font><font color="#ff0000">$zone_file</font>
<b><font color="#ffffff">fi</font></b>
<font color="#ff0000">cat </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#F3E651">|</font><font color="#ff0000"> transform </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font>
<font color="#ff0000">grep -v </font><font color="#bb00ff">' ; serial'</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp</font>
<font color="#ff0000">grep -v </font><font color="#bb00ff">' ; serial'</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#F3E651">></font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">old</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp</font>
<font color="#ff0000">echo </font><font color="#bb00ff">"Has zone $zone_file changed?"</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> diff -u </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">old</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">noserial</font><font color="#F3E651">.</font><font color="#ff0000">tmp</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"The zone $zone_file hasn't changed"</font>
<font color="#ff0000"> rm </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.*.</font><font color="#ff0000">tmp</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> </font><font color="#bb00ff">0</font>
<b><font color="#ffffff">fi</font></b>
<font color="#ff0000">cp </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak</font>
<font color="#ff0000">mv </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">new</font><font color="#F3E651">.</font><font color="#ff0000">tmp </font><font color="#ff0000">$zone_file</font>
<font color="#ff0000">rm </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.*.</font><font color="#ff0000">tmp</font>
<font color="#ff0000">echo </font><font color="#bb00ff">"Reloading nsd"</font>
<font color="#ff0000">nsd-control reload</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">!</font><font color="#ff0000"> zone_is_ok </font><font color="#ff0000">$zone</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> echo </font><font color="#bb00ff">"Rolling back $zone_file changes"</font>
<font color="#ff0000"> cp </font><font color="#ff0000">$zone_file</font><font color="#ff0000"> </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">invalid</font>
<font color="#ff0000"> mv </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">bak </font><font color="#ff0000">$zone_file</font>
<font color="#ff0000"> echo </font><font color="#bb00ff">"Reloading nsd"</font>
<font color="#ff0000"> nsd-control reload</font>
<font color="#ff0000"> zone_is_ok </font><font color="#ff0000">$zone</font>
<font color="#ff0000"> </font><b><font color="#ffffff">return</font></b><font color="#ff0000"> </font><font color="#bb00ff">3</font>
<b><font color="#ffffff">fi</font></b>
<b><font color="#ffffff">for</font></b><font color="#ff0000"> cleanup </font><b><font color="#ffffff">in</font></b><font color="#ff0000"> invalid bak</font><font color="#F3E651">;</font><font color="#ff0000"> </font><b><font color="#ffffff">do</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> -f </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">$cleanup</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> rm </font><font color="#ff0000">$zone_file</font><font color="#F3E651">.</font><font color="#ff0000">$cleanup</font>
<font color="#ff0000"> </font><b><font color="#ffffff">fi</font></b>
<b><font color="#ffffff">done</font></b>
<font color="#ff0000">echo </font><font color="#bb00ff">"Failover of zone $zone to $MASTER completed"</font>
<b><font color="#ffffff">return</font></b><font color="#ff0000"> </font><font color="#bb00ff">1</font>
</pre>
<br />
<span>A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output.</span><br />
<br />
<span>The authorative nameserver for my domains runs on both VMs, and both are configured to be a "master" DNS server so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case.</span><br />
<br />
<span>Check out the whole script here:</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/frontends/scripts/dns-failover.ksh'>dns-failover.ksh</a><br />
<br />
<h3 style='display: inline' id='fairly-cheap-and-geo-redundant'>Fairly cheap and geo-redundant</h3><br />
<br />
<span>I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany).</span><br />
<br />
<a class='textlink' href='https://OpenBSD.Amsterdam'>https://OpenBSD.Amsterdam</a><br />
<a class='textlink' href='https://www.Hetzner.cloud'>https://www.Hetzner.cloud</a><br />
<br />
<span>I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required.</span><br />
<br />
<span>A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions.</span><br />
<br />
<h3 style='display: inline' id='failover-time-and-split-brain'>Failover time and split-brain</h3><br />
<br />
<span>A DNS failover doesn't happen immediately. I've configured a DNS TTL of <span class='inlinecode'>300</span> seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover).</span><br />
<br />
<span>A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated.</span><br />
<br />
<h3 style='display: inline' id='failover-support-for-multiple-protocols'>Failover support for multiple protocols</h3><br />
<br />
<span>With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's <span class='inlinecode'>httpd</span> (OpenBSD's HTTP server) and <span class='inlinecode'>relayd</span> (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entries, which VM receives the requests.</span><br />
<br />
<a class='textlink' href='https://man.OpenBSD.org/httpd.8'>https://man.OpenBSD.org/httpd.8</a><br />
<a class='textlink' href='https://man.OpenBSD.org/relayd.8'>https://man.OpenBSD.org/relayd.8</a><br />
<br />
<span>For example, the master is responsible for the <span class='inlinecode'>https://www.foo.zone</span> and <span class='inlinecode'>https://foo.zone</span> hosts, whereas the standby can be reached via <span class='inlinecode'>https://standby.foo.zone</span> (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. <span class='inlinecode'>irregular.ninja</span>, <span class='inlinecode'>paul.buetow.org</span> and so on. The same applies to my Gemini capsules for <span class='inlinecode'>gemini://foo.zone</span>, <span class='inlinecode'>gemini://standby.foo.zone</span>, <span class='inlinecode'>gemini://paul.buetow.org</span> and <span class='inlinecode'>gemini://standby.paul.buetow.org</span>.</span><br />
<br />
<span>On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)!</span><br />
<br />
<h3 style='display: inline' id='let-s-encrypt-tls-certificates'>Let's encrypt TLS certificates</h3><br />
<br />
<span>All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate.</span><br />
<br />
<span>If the master always serves <span class='inlinecode'>foo.zone</span> and the standby always <span class='inlinecode'>standby.foo.zone</span>, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for <span class='inlinecode'>foo.zone</span> and the new standby wouldn't have a valid certificate for <span class='inlinecode'>standby.foo.zone</span> which would lead to TLS errors on the clients.</span><br />
<br />
<span>As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that:</span><br />
<br />
<ul>
<li>In an odd week number, the first server is the default master</li>
<li>In an even week number, the second server is the default master.</li>
</ul><br />
<span>Which translates to:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><i><font color="#ababab"># Weekly auto-failover for Let's Encrypt automation</font></i>
<b><font color="#ffffff">local</font></b><font color="#ff0000"> -i -r </font><font color="#ff0000">week_of_the_year</font><font color="#F3E651">=</font><font color="#ff0000">$(</font><font color="#ff0000">date </font><font color="#F3E651">+%</font><font color="#ff0000">U</font><font color="#F3E651">)</font>
<b><font color="#ffffff">if</font></b><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000"> </font><font color="#ff0000">$(</font><font color="#F3E651">(</font><font color="#ff0000"> week_of_the_year </font><font color="#F3E651">%</font><font color="#ff0000"> </font><font color="#bb00ff">2</font><font color="#ff0000"> </font><font color="#F3E651">))</font><font color="#ff0000"> -eq </font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#F3E651">];</font><font color="#ff0000"> </font><b><font color="#ffffff">then</font></b>
<font color="#ff0000"> </font><b><font color="#ffffff">local</font></b><font color="#ff0000"> </font><font color="#ff0000">tmp</font><font color="#F3E651">=</font><font color="#ff0000">$master</font>
<font color="#ff0000"> </font><font color="#ff0000">master</font><font color="#F3E651">=</font><font color="#ff0000">$standby</font>
<font color="#ff0000"> </font><font color="#ff0000">standby</font><font color="#F3E651">=</font><font color="#ff0000">$tmp</font>
<b><font color="#ffffff">fi</font></b>
</pre>
<br />
<span>This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM.</span><br />
<br />
<span>The ACME automation is yet another daily CRON script <span class='inlinecode'>/usr/local/bin/acme.sh</span>. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt. </span><br />
<br />
<span>Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty.</span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/frontends/scripts/acme.sh.tpl'><span class='inlinecode'>acme.sh.tpl</span> - Rex template for the <span class='inlinecode'>acme.sh</span> script of mine.</a><br />
<a class='textlink' href='https://man.OpenBSD.org/acme-client.1'>https://man.OpenBSD.org/acme-client.1</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's Encrypt with OpenBSD and Rex</a><br />
<br />
<h3 style='display: inline' id='monitoring'>Monitoring</h3><br />
<br />
<span>CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed. </span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/gogios'>https://codeberg.org/snonux/gogios</a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>KISS server monitoring with Gogios</a><br />
<br />
<span>Gogios, as I developed it by myself, isn't part of the OpenBSD base system. </span><br />
<br />
<h3 style='display: inline' id='rex-automation'>Rex automation</h3><br />
<br />
<span>I use Rexify, a friendly configuration management system that allows automatic deployment and configuration.</span><br />
<br />
<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br />
<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/frontends'>codeberg.org/snonux/rexfiles/frontends</a><br />
<br />
<span>Rex isn't part of the OpenBSD base system, but I didn't need to install any external software on OpenBSD either as Rex is invoked from my Laptop!</span><br />
<br />
<h2 style='display: inline' id='more-ha'>More HA</h2><br />
<br />
<span>Other high-available services running on my OpenBSD VMs are my MTAs for mail forwarding (OpenSMTPD - also part of the OpenBSD base system) and the authoritative DNS servers (<span class='inlinecode'>nsd</span>) for all my domains. No particular HA setup is required, though, as the protocols (SMTP and DNS) already take care of the failover to the next available host! </span><br />
<br />
<a class='textlink' href='https://www.OpenSMTPD.org/'>https://www.OpenSMTPD.org/</a><br />
<br />
<span>As a password manager, I use <span class='inlinecode'>geheim</span>, a command-line tool I wrote in Ruby with encrypted files in a git repository (I even have it installed in Termux on my Phone). For HA reasons, I simply updated the client code so that it always synchronises the database with both servers when I run the <span class='inlinecode'>sync</span> command there. </span><br />
<br />
<a class='textlink' href='https://codeberg.org/snonux/geheim'>https://codeberg.org/snonux/geheim</a><br />
<br />
<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br />
<br />
<span>Other *BSD and KISS related posts are:</span><br />
<br />
<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br />
<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br />
<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD (You are currently reading this)</a><br />
<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br />
<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br />
<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios</a><br />
<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br />
<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br />
<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br />
<br />
<a class='textlink' href='../'>Back to the main site</a><br />
<p class="footer">
Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
<br />
Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a>
</p>
<script type="text/javascript" src="../retrofuturistic.js"></script>
</body>
</html>
|
