diff options
| -rw-r--r-- | gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html | 322 | ||||
| -rw-r--r-- | gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.html | 320 | ||||
| -rw-r--r-- | gemfeed/atom.xml | 797 | ||||
| -rw-r--r-- | gemfeed/index.html | 1 | ||||
| -rw-r--r-- | index.html | 3 | ||||
| -rw-r--r-- | projects.html | 26 | ||||
| -rw-r--r-- | uptime-stats.html | 2 |
7 files changed, 985 insertions, 486 deletions
diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html new file mode 100644 index 00000000..d9d8cfe0 --- /dev/null +++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html @@ -0,0 +1,322 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<title>KISS high-availability with OpenBSD</title> +<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> +<link rel="stylesheet" href="../style.css" /> +<link rel="stylesheet" href="style-override.css" /> +</head> +<body> +<h1 style='display: inline'>KISS high-availability with OpenBSD</h1><br /> +<br /> +<span class='quote'>Published at 2024-03-30T22:12:56+02:00</span><br /> +<br /> +<pre> +Art by Michael J. Penick (mod. by Paul B) + + __________ + / nsd tower\ ( + /____________\ (\) awk-ward + |:_:_:_:_:_| )) plant + |_:_,--.:_:| dig-bubble (\// ) + |:_:|__|_:_| relayd-castle _ ) )) (( + _ |_ _ :_:| _ _ _ (_) (((( /)\` + | |_| |_| | _| | |_| |_| | o \\)) (( ( + \_:_:_:_:/|_|_|_|\:_:_:_:_/ . (( )))) + |_,-._:_:_:_:_:_:_:_.-,_| )) ((// + |:|_|:_:_:,---,:_:_:|_|:| ,-. )/ + |_:_:_:_,'puffy `,_:_:_:_| _ o ,;'))(( + |:_:_:_/ _ | _ \_:_:_:| (_O (( )) +_____|_:_:_| (o)-(o) |_:_:_|--'`-. ,--. ksh under-water (((\'/ + ', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\ /,---.~ goat \`)) +. ` |_:_:_| \`-'/ |_:_:_|. ` . ` /()\.__( ) .,-----'`-\(( sed-root + ', ;|:_:_:| `-' |:_:_:| ', ; ', ; `--'| \ ', ; ', ; ',')).,-- +. ` MJP ` . ` . ` . ` . httpd-soil ` . . ` . ` . ` . ` . ` + ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; + +</pre> +<br /> +<span>I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like BGP, LVS/IPVS, ldirectord, Pacemaker, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work. </span><br /> +<br /> +<span>But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space—something as KISS (keep it simple and stupid) as possible.</span><br /> +<br /> +<span>It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway.</span><br /> +<br /> +<span class='quote'>PS: ASCII-art reflects the OpenBSD under-water world with all the tools available in the base system.</span><br /> +<br /> +<h2 style='display: inline'>My auto-failover requirements</h2><br /> +<br /> +<ul> +<li>Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible. </li> +<li>Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month).</li> +<li>It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.</li> +<li>It should be geo-redundant. </li> +<li>It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.</li> +<li>Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.</li> +<li>Let's Encrypt TLS certificates should always work (before and after a failover).</li> +<li>Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover.</li> +<li>Don't configure everything manually. The configuration should be automated and reproducible.</li> +</ul><br /> +<h2 style='display: inline'>My HA solution</h2><br /> +<br /> +<h3 style='display: inline'>Only OpenBSD base installation required</h3><br /> +<br /> +<span>My HA solution for Web and Gemini is based on DNS (OpenBSD's <span class='inlinecode'>nsd</span>) and a simple shell script (OpenBSD's <span class='inlinecode'>ksh</span> and some little <span class='inlinecode'>sed</span> and <span class='inlinecode'>awk</span> and <span class='inlinecode'>grep</span>). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system.</span><br /> +<br /> +<a class='textlink' href='https://man.OpenBSD.org/nsd.8'>https://man.OpenBSD.org/nsd.8</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/ksh'>https://man.OpenBSD.org/ksh</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/awk'>https://man.OpenBSD.org/awk</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/sed'>https://man.OpenBSD.org/sed</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/dig'>https://man.OpenBSD.org/dig</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/ftp'>https://man.OpenBSD.org/ftp</a><br /> +<br /> +<span>I also used the <span class='inlinecode'>dig</span> (for DNS checks) and <span class='inlinecode'>ftp</span> (for HTTP/HTTPS checks) programs. </span><br /> +<br /> +<span>The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The <span class='inlinecode'>ksh</span> script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM: </span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900">#!/bin/ksh</font></i> + +<font color="#009900">ZONES_DIR</font><font color="#990000">=</font>/var/nsd/zones/master<font color="#990000">/</font> +<font color="#009900">DEFAULT_MASTER</font><font color="#990000">=</font>fishfinger<font color="#990000">.</font>buetow<font color="#990000">.</font>org +<font color="#009900">DEFAULT_STANDBY</font><font color="#990000">=</font>blowfish<font color="#990000">.</font>buetow<font color="#990000">.</font>org + +<b><font color="#000000">determine_master_and_standby ()</font></b> { + <b><font color="#0000FF">local</font></b> <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$DEFAULT_MASTER</font> + <b><font color="#0000FF">local</font></b> <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$DEFAULT_STANDBY</font> + + <font color="#990000">.</font> + <font color="#990000">.</font> + <font color="#990000">.</font> + + <b><font color="#0000FF">local</font></b> -i <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">1</font> + <b><font color="#0000FF">if</font></b> <font color="#990000">!</font> ftp -<font color="#993399">4</font> -o - https<font color="#990000">://</font><font color="#009900">$master</font>/index<font color="#990000">.</font>txt <font color="#990000">|</font> grep -q <font color="#FF0000">"Welcome to $master"</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"https://$master/index.txt IPv4 health check failed"</font> + <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">0</font> + <b><font color="#0000FF">elif</font></b> <font color="#990000">!</font> ftp -<font color="#993399">6</font> -o - https<font color="#990000">://</font><font color="#009900">$master</font>/index<font color="#990000">.</font>txt <font color="#990000">|</font> grep -q <font color="#FF0000">"Welcome to $master"</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"https://$master/index.txt IPv6 health check failed"</font> + <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">0</font> + <b><font color="#0000FF">fi</font></b> + <b><font color="#0000FF">if</font></b> <font color="#990000">[</font> <font color="#009900">$health_ok</font> -eq <font color="#993399">0</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + <b><font color="#0000FF">local</font></b> <font color="#009900">tmp</font><font color="#990000">=</font><font color="#009900">$master</font> + <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$standby</font> + <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$tmp</font> + <b><font color="#0000FF">fi</font></b> + + <font color="#990000">.</font> + <font color="#990000">.</font> + <font color="#990000">.</font> +} +</pre> +<br /> +<span>The failover scripts looks for the <span class='inlinecode'> ; Enable failover</span> string in the DNS zone files and swaps the <span class='inlinecode'>A</span> and <span class='inlinecode'>AAAA</span> records of the DNS entries accordingly:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>fishfinger$ grep failover /var/nsd/zones/master/foo<font color="#990000">.</font>zone<font color="#990000">.</font>zone + <font color="#993399">300</font> IN A <font color="#993399">46.23</font><font color="#990000">.</font><font color="#993399">94.99</font> <font color="#990000">;</font> Enable failover + <font color="#993399">300</font> IN AAAA 2a03<font color="#990000">:</font><font color="#993399">6000</font><font color="#990000">:</font>6f67<font color="#990000">:</font><font color="#993399">624</font><font color="#990000">::</font><font color="#993399">99</font> <font color="#990000">;</font> Enable failover +www <font color="#993399">300</font> IN A <font color="#993399">46.23</font><font color="#990000">.</font><font color="#993399">94.99</font> <font color="#990000">;</font> Enable failover +www <font color="#993399">300</font> IN AAAA 2a03<font color="#990000">:</font><font color="#993399">6000</font><font color="#990000">:</font>6f67<font color="#990000">:</font><font color="#993399">624</font><font color="#990000">::</font><font color="#993399">99</font> <font color="#990000">;</font> Enable failover +standby <font color="#993399">300</font> IN A <font color="#993399">23.88</font><font color="#990000">.</font><font color="#993399">35.144</font> <font color="#990000">;</font> Enable failover +standby <font color="#993399">300</font> IN AAAA 2a01<font color="#990000">:</font>4f8<font color="#990000">:</font>c17<font color="#990000">:</font>20f1<font color="#990000">::</font><font color="#993399">42</font> <font color="#990000">;</font> Enable failover +</pre> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><b><font color="#000000">tramsform ()</font></b> { + sed -E <font color="#FF0000">'</font> +<font color="#FF0000"> /IN A .*; Enable failover/ {</font> +<font color="#FF0000"> /^standby/! {</font> +<font color="#FF0000"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN A '</font><font color="#009900">$(</font>cat /var/nsd/run/master_a<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /^standby/ {</font> +<font color="#FF0000"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN A '</font><font color="#009900">$(</font>cat /var/nsd/run/standby_a<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /IN AAAA .*; Enable failover/ {</font> +<font color="#FF0000"> /^standby/! {</font> +<font color="#FF0000"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN AAAA '</font><font color="#009900">$(</font>cat /var/nsd/run/master_aaaa<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /^standby/ {</font> +<font color="#FF0000"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN AAAA '</font><font color="#009900">$(</font>cat /var/nsd/run/standby_aaaa<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> / ; serial/ {</font> +<font color="#FF0000"> s/^( +) ([0-9]+) .*; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> '</font><font color="#009900">$(</font>date <font color="#990000">+%</font>s<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> '</font> +} +</pre> +<br /> +<span>After the failover, the script reloads <span class='inlinecode'>nsd</span> and performs a sanity check to see if DNS still works. If not, a rollback will be performed:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900"># Race condition (e.g. script execution abored in the middle of the previous run)</font></i> +<b><font color="#0000FF">if</font></b> <font color="#990000">[</font> -f <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + mv <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#009900">$zone_file</font> +<b><font color="#0000FF">fi</font></b> + +cat <font color="#009900">$zone_file</font> <font color="#990000">|</font> transform <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp + +grep -v <font color="#FF0000">' ; serial'</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp +grep -v <font color="#FF0000">' ; serial'</font> <font color="#009900">$zone_file</font> <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>old<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp + +echo <font color="#FF0000">"Has zone $zone_file changed?"</font> +<b><font color="#0000FF">if</font></b> diff -u <font color="#009900">$zone_file</font><font color="#990000">.</font>old<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp<font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"The zone $zone_file hasn't changed"</font> + rm <font color="#009900">$zone_file</font><font color="#990000">.*.</font>tmp + <b><font color="#0000FF">return</font></b> <font color="#993399">0</font> +<b><font color="#0000FF">fi</font></b> + +cp <font color="#009900">$zone_file</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>bak +mv <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp <font color="#009900">$zone_file</font> +rm <font color="#009900">$zone_file</font><font color="#990000">.*.</font>tmp +echo <font color="#FF0000">"Reloading nsd"</font> +nsd-control reload + +<b><font color="#0000FF">if</font></b> <font color="#990000">!</font> zone_is_ok <font color="#009900">$zone</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"Rolling back $zone_file changes"</font> + cp <font color="#009900">$zone_file</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>invalid + mv <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#009900">$zone_file</font> + echo <font color="#FF0000">"Reloading nsd"</font> + nsd-control reload + zone_is_ok <font color="#009900">$zone</font> + <b><font color="#0000FF">return</font></b> <font color="#993399">3</font> +<b><font color="#0000FF">fi</font></b> + +<b><font color="#0000FF">for</font></b> cleanup <b><font color="#0000FF">in</font></b> invalid bak<font color="#990000">;</font> <b><font color="#0000FF">do</font></b> + <b><font color="#0000FF">if</font></b> <font color="#990000">[</font> -f <font color="#009900">$zone_file</font><font color="#990000">.</font><font color="#009900">$cleanup</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + rm <font color="#009900">$zone_file</font><font color="#990000">.</font><font color="#009900">$cleanup</font> + <b><font color="#0000FF">fi</font></b> +<b><font color="#0000FF">done</font></b> + +echo <font color="#FF0000">"Failover of zone $zone to $MASTER completed"</font> +<b><font color="#0000FF">return</font></b> <font color="#993399">1</font> +</pre> +<br /> +<span>A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output.</span><br /> +<br /> +<span>The nameserver is running on both VMs, and both are configured to be "master" DNS servers so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case.</span><br /> +<br /> +<span>Check out the whole script here:</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh'>https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh</a><br /> +<br /> +<h3 style='display: inline'>Fairly cheap and geo-redundant</h3><br /> +<br /> +<span>I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany).</span><br /> +<br /> +<a class='textlink' href='https://openbsd.amsterdam'>https://openbsd.amsterdam</a><br /> +<a class='textlink' href='https://www.hetzner.cloud'>https://www.hetzner.cloud</a><br /> +<br /> +<span>I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required.</span><br /> +<br /> +<span>A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions.</span><br /> +<br /> +<h3 style='display: inline'>Failover time and split-brain</h3><br /> +<br /> +<span>A DNS failover doesn't happen immediately. I've configured a DNS TTL of <span class='inlinecode'>300</span> seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover).</span><br /> +<br /> +<span>A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated.</span><br /> +<br /> +<h3 style='display: inline'>Failover support for multiple protocols</h3><br /> +<br /> +<span>With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's <span class='inlinecode'>httpd</span> (OpenBSD's HTTP server) and <span class='inlinecode'>relayd</span> (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entry, which hosts receive the requests.</span><br /> +<br /> +<a class='textlink' href='https://man.openbsd.org/httpd.8'>https://man.openbsd.org/httpd.8</a><br /> +<a class='textlink' href='https://man.openbsd.org/relayd.8'>https://man.openbsd.org/relayd.8</a><br /> +<br /> +<span>For example, the master is responsible for the <span class='inlinecode'>https://www.foo.zone</span> and <span class='inlinecode'>https://foo.zone</span> hosts, whereas the standby can be reached via <span class='inlinecode'>https://standby.foo.zone</span> (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. <span class='inlinecode'>irregular.ninja</span>, <span class='inlinecode'>paul.buetow.org</span> and so on. The same applies to my Gemini capsules for <span class='inlinecode'>gemini://foo.zone</span>, <span class='inlinecode'>gemini://standby.foo.zone</span>, <span class='inlinecode'>gemini://paul.buetow.org</span> and <span class='inlinecode'>gemini://standby.paul.buetow.org</span>.</span><br /> +<br /> +<span>On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)!</span><br /> +<br /> +<h3 style='display: inline'>Let's encrypt TLS certificates</h3><br /> +<br /> +<span>All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate.</span><br /> +<br /> +<span>If the master always serves <span class='inlinecode'>foo.zone</span> and the standby always <span class='inlinecode'>standby.foo.zone</span>, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for <span class='inlinecode'>foo.zone</span> and the new standby wouldn't have a valid certificate for <span class='inlinecode'>standby.foo.zone</span> which would lead to TLS errors on the clients.</span><br /> +<br /> +<span>As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that:</span><br /> +<br /> +<ul> +<li>In an odd week number, the first server is the default master</li> +<li>In an even week number, the second server is the default master.</li> +</ul><br /> +<span>Which translates to:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900"># Weekly auto-failover for Let's Encrypt automation</font></i> +<b><font color="#0000FF">local</font></b> -i -r <font color="#009900">week_of_the_year</font><font color="#990000">=</font><font color="#009900">$(</font>date <font color="#990000">+%</font>U<font color="#990000">)</font> +<b><font color="#0000FF">if</font></b> <font color="#990000">[</font> <font color="#009900">$(</font><font color="#990000">(</font> week_of_the_year <font color="#990000">%</font> <font color="#993399">2</font> <font color="#990000">))</font> -eq <font color="#993399">0</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + <b><font color="#0000FF">local</font></b> <font color="#009900">tmp</font><font color="#990000">=</font><font color="#009900">$master</font> + <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$standby</font> + <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$tmp</font> +<b><font color="#0000FF">fi</font></b> +</pre> +<br /> +<span>This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM.</span><br /> +<br /> +<span>The ACME automation is yet another daily CRON script <span class='inlinecode'>/usr/local/bin/acme.sh</span>. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt. </span><br /> +<br /> +<span>Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty.</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/acme.sh.tpl'><span class='inlinecode'>acme.sh.tpl</span> - Rex template for the <span class='inlinecode'>acme.sh</span> script of mine.</a><br /> +<a class='textlink' href='https://man.openbsd.org/acme-client.1'>https://man.openbsd.org/acme-client.1</a><br /> +<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's Encrypt with OpenBSD and Rex</a><br /> +<br /> +<h3 style='display: inline'>Monitoring</h3><br /> +<br /> +<span>CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed. </span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/gogios'>https://codeberg.org/snonux/gogios</a><br /> +<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>KISS server monitoring with Gogios</a><br /> +<span> </span><br /> +<h3 style='display: inline'>Rex automation</h3><br /> +<br /> +<span>I use Rexify, a friendly configuration management system that allows automatic deployment and configuration.</span><br /> +<br /> +<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends'>https://codeberg.org/snonux/rexfiles/src/branch/master/frontends</a><br /> +<br /> +<h2 style='display: inline'>More HA</h2><br /> +<br /> +<span>Other high-available services running on my OpenBSD VMs are my MTAs for mail forwarding (OpenSMTPD) and the authoritative DNS servers (<span class='inlinecode'>nsd</span>) for all my domains. No particular HA setup is required, though, as the protocols (SMTP and DNS) already take care of the failover to the next available host! </span><br /> +<br /> +<span>As a password manager, I use <span class='inlinecode'>geheim</span>, a command-line tool I wrote in Ruby with encrypted files in a git repository (I even have it installed in Termux on my Phone). For HA reasons, I simply updated the client code so that it always synchronises the database with both servers when I run the <span class='inlinecode'>sync</span> command there. </span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/geheim'>https://codeberg.org/snonux/geheim</a><br /> +<br /> +<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> +<br /> +<span>Other *BSD and KISS related posts are:</span><br /> +<br /> +<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br /> +<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br /> +<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br /> +<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios</a><br /> +<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br /> +<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br /> +<br /> +<a class='textlink' href='../'>Back to the main site</a><br /> +<p class="footer"> +Generated by <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 2.1.0-release</a> | +served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | +<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a> +</p> +</body> +</html> diff --git a/gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.html b/gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.html new file mode 100644 index 00000000..6d5916af --- /dev/null +++ b/gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.html @@ -0,0 +1,320 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<title>KISS high-availability with OpenBSD</title> +<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> +<link rel="stylesheet" href="../style.css" /> +<link rel="stylesheet" href="style-override.css" /> +</head> +<body> +<h1 style='display: inline'>KISS high-availability with OpenBSD</h1><br /> +<br /> +<pre> +Art by Michael J. Penick (mod. by Paul B) + + __________ + / nsd tower\ ( + /____________\ (\) awk-ward + |:_:_:_:_:_| )) plant + |_:_,--.:_:| dig-bubble (\// ) + |:_:|__|_:_| relayd-castle _ ) )) (( + _ |_ _ :_:| _ _ _ (_) (((( /)\` + | |_| |_| | _| | |_| |_| | o \\)) (( ( + \_:_:_:_:/|_|_|_|\:_:_:_:_/ . (( )))) + |_,-._:_:_:_:_:_:_:_.-,_| )) ((// + |:|_|:_:_:,---,:_:_:|_|:| ,-. )/ + |_:_:_:_,'puffy `,_:_:_:_| _ o ,;'))(( + |:_:_:_/ _ | _ \_:_:_:| (_O (( )) +_____|_:_:_| (o)-(o) |_:_:_|--'`-. ,--. ksh under-water (((\'/ + ', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\ /,---.~ goat \`)) +. ` |_:_:_| \`-'/ |_:_:_|. ` . ` /()\.__( ) .,-----'`-\(( sed-root + ', ;|:_:_:| `-' |:_:_:| ', ; ', ; `--'| \ ', ; ', ; ',')).,-- +. ` MJP ` . ` . ` . ` . httpd-soil ` . . ` . ` . ` . ` . ` + ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; + +</pre> +<br /> +<span>I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like BGP, LVS/IPVS, ldirectord, Pacemaker, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work. </span><br /> +<br /> +<span>But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space—something as KISS (keep it simple and stupid) as possible.</span><br /> +<br /> +<span>It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway.</span><br /> +<br /> +<span class='quote'>PS: ASCII-art reflects the OpenBSD under-water world with all the tools available in the base system.</span><br /> +<br /> +<h2 style='display: inline'>My auto-failover requirements</h2><br /> +<br /> +<ul> +<li>Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible. </li> +<li>Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month).</li> +<li>It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.</li> +<li>It should be geo-redundant. </li> +<li>It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.</li> +<li>Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.</li> +<li>Let's Encrypt TLS certificates should always work (before and after a failover).</li> +<li>Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover.</li> +<li>Don't configure everything manually. The configuration should be automated and reproducible.</li> +</ul><br /> +<h2 style='display: inline'>My HA solution</h2><br /> +<br /> +<h3 style='display: inline'>Only OpenBSD base installation required</h3><br /> +<br /> +<span>My HA solution for Web and Gemini is based on DNS (OpenBSD's <span class='inlinecode'>nsd</span>) and a simple shell script (OpenBSD's <span class='inlinecode'>ksh</span> and some little <span class='inlinecode'>sed</span> and <span class='inlinecode'>awk</span> and <span class='inlinecode'>grep</span>). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system.</span><br /> +<br /> +<a class='textlink' href='https://man.OpenBSD.org/nsd.8'>https://man.OpenBSD.org/nsd.8</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/ksh'>https://man.OpenBSD.org/ksh</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/awk'>https://man.OpenBSD.org/awk</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/sed'>https://man.OpenBSD.org/sed</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/dig'>https://man.OpenBSD.org/dig</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/ftp'>https://man.OpenBSD.org/ftp</a><br /> +<br /> +<span>I also used the <span class='inlinecode'>dig</span> (for DNS checks) and <span class='inlinecode'>ftp</span> (for HTTP/HTTPS checks) programs. </span><br /> +<br /> +<span>The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The <span class='inlinecode'>ksh</span> script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM: </span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900">#!/bin/ksh</font></i> + +<font color="#009900">ZONES_DIR</font><font color="#990000">=</font>/var/nsd/zones/master<font color="#990000">/</font> +<font color="#009900">DEFAULT_MASTER</font><font color="#990000">=</font>fishfinger<font color="#990000">.</font>buetow<font color="#990000">.</font>org +<font color="#009900">DEFAULT_STANDBY</font><font color="#990000">=</font>blowfish<font color="#990000">.</font>buetow<font color="#990000">.</font>org + +<b><font color="#000000">determine_master_and_standby ()</font></b> { + <b><font color="#0000FF">local</font></b> <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$DEFAULT_MASTER</font> + <b><font color="#0000FF">local</font></b> <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$DEFAULT_STANDBY</font> + + <font color="#990000">.</font> + <font color="#990000">.</font> + <font color="#990000">.</font> + + <b><font color="#0000FF">local</font></b> -i <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">1</font> + <b><font color="#0000FF">if</font></b> <font color="#990000">!</font> ftp -<font color="#993399">4</font> -o - https<font color="#990000">://</font><font color="#009900">$master</font>/index<font color="#990000">.</font>txt <font color="#990000">|</font> grep -q <font color="#FF0000">"Welcome to $master"</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"https://$master/index.txt IPv4 health check failed"</font> + <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">0</font> + <b><font color="#0000FF">elif</font></b> <font color="#990000">!</font> ftp -<font color="#993399">6</font> -o - https<font color="#990000">://</font><font color="#009900">$master</font>/index<font color="#990000">.</font>txt <font color="#990000">|</font> grep -q <font color="#FF0000">"Welcome to $master"</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"https://$master/index.txt IPv6 health check failed"</font> + <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">0</font> + <b><font color="#0000FF">fi</font></b> + <b><font color="#0000FF">if</font></b> <font color="#990000">[</font> <font color="#009900">$health_ok</font> -eq <font color="#993399">0</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + <b><font color="#0000FF">local</font></b> <font color="#009900">tmp</font><font color="#990000">=</font><font color="#009900">$master</font> + <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$standby</font> + <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$tmp</font> + <b><font color="#0000FF">fi</font></b> + + <font color="#990000">.</font> + <font color="#990000">.</font> + <font color="#990000">.</font> +} +</pre> +<br /> +<span>The failover scripts looks for the <span class='inlinecode'> ; Enable failover</span> string in the DNS zone files and swaps the <span class='inlinecode'>A</span> and <span class='inlinecode'>AAAA</span> records of the DNS entries accordingly:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>fishfinger$ grep failover /var/nsd/zones/master/foo<font color="#990000">.</font>zone<font color="#990000">.</font>zone + <font color="#993399">300</font> IN A <font color="#993399">46.23</font><font color="#990000">.</font><font color="#993399">94.99</font> <font color="#990000">;</font> Enable failover + <font color="#993399">300</font> IN AAAA 2a03<font color="#990000">:</font><font color="#993399">6000</font><font color="#990000">:</font>6f67<font color="#990000">:</font><font color="#993399">624</font><font color="#990000">::</font><font color="#993399">99</font> <font color="#990000">;</font> Enable failover +www <font color="#993399">300</font> IN A <font color="#993399">46.23</font><font color="#990000">.</font><font color="#993399">94.99</font> <font color="#990000">;</font> Enable failover +www <font color="#993399">300</font> IN AAAA 2a03<font color="#990000">:</font><font color="#993399">6000</font><font color="#990000">:</font>6f67<font color="#990000">:</font><font color="#993399">624</font><font color="#990000">::</font><font color="#993399">99</font> <font color="#990000">;</font> Enable failover +standby <font color="#993399">300</font> IN A <font color="#993399">23.88</font><font color="#990000">.</font><font color="#993399">35.144</font> <font color="#990000">;</font> Enable failover +standby <font color="#993399">300</font> IN AAAA 2a01<font color="#990000">:</font>4f8<font color="#990000">:</font>c17<font color="#990000">:</font>20f1<font color="#990000">::</font><font color="#993399">42</font> <font color="#990000">;</font> Enable failover +</pre> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><b><font color="#000000">tramsform ()</font></b> { + sed -E <font color="#FF0000">'</font> +<font color="#FF0000"> /IN A .*; Enable failover/ {</font> +<font color="#FF0000"> /^standby/! {</font> +<font color="#FF0000"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN A '</font><font color="#009900">$(</font>cat /var/nsd/run/master_a<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /^standby/ {</font> +<font color="#FF0000"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN A '</font><font color="#009900">$(</font>cat /var/nsd/run/standby_a<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /IN AAAA .*; Enable failover/ {</font> +<font color="#FF0000"> /^standby/! {</font> +<font color="#FF0000"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN AAAA '</font><font color="#009900">$(</font>cat /var/nsd/run/master_aaaa<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /^standby/ {</font> +<font color="#FF0000"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN AAAA '</font><font color="#009900">$(</font>cat /var/nsd/run/standby_aaaa<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> / ; serial/ {</font> +<font color="#FF0000"> s/^( +) ([0-9]+) .*; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> '</font><font color="#009900">$(</font>date <font color="#990000">+%</font>s<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> '</font> +} +</pre> +<br /> +<span>After the failover, the script reloads <span class='inlinecode'>nsd</span> and performs a sanity check to see if DNS still works. If not, a rollback will be performed:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900"># Race condition (e.g. script execution abored in the middle of the previous run)</font></i> +<b><font color="#0000FF">if</font></b> <font color="#990000">[</font> -f <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + mv <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#009900">$zone_file</font> +<b><font color="#0000FF">fi</font></b> + +cat <font color="#009900">$zone_file</font> <font color="#990000">|</font> transform <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp + +grep -v <font color="#FF0000">' ; serial'</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp +grep -v <font color="#FF0000">' ; serial'</font> <font color="#009900">$zone_file</font> <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>old<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp + +echo <font color="#FF0000">"Has zone $zone_file changed?"</font> +<b><font color="#0000FF">if</font></b> diff -u <font color="#009900">$zone_file</font><font color="#990000">.</font>old<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp<font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"The zone $zone_file hasn't changed"</font> + rm <font color="#009900">$zone_file</font><font color="#990000">.*.</font>tmp + <b><font color="#0000FF">return</font></b> <font color="#993399">0</font> +<b><font color="#0000FF">fi</font></b> + +cp <font color="#009900">$zone_file</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>bak +mv <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp <font color="#009900">$zone_file</font> +rm <font color="#009900">$zone_file</font><font color="#990000">.*.</font>tmp +echo <font color="#FF0000">"Reloading nsd"</font> +nsd-control reload + +<b><font color="#0000FF">if</font></b> <font color="#990000">!</font> zone_is_ok <font color="#009900">$zone</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"Rolling back $zone_file changes"</font> + cp <font color="#009900">$zone_file</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>invalid + mv <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#009900">$zone_file</font> + echo <font color="#FF0000">"Reloading nsd"</font> + nsd-control reload + zone_is_ok <font color="#009900">$zone</font> + <b><font color="#0000FF">return</font></b> <font color="#993399">3</font> +<b><font color="#0000FF">fi</font></b> + +<b><font color="#0000FF">for</font></b> cleanup <b><font color="#0000FF">in</font></b> invalid bak<font color="#990000">;</font> <b><font color="#0000FF">do</font></b> + <b><font color="#0000FF">if</font></b> <font color="#990000">[</font> -f <font color="#009900">$zone_file</font><font color="#990000">.</font><font color="#009900">$cleanup</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + rm <font color="#009900">$zone_file</font><font color="#990000">.</font><font color="#009900">$cleanup</font> + <b><font color="#0000FF">fi</font></b> +<b><font color="#0000FF">done</font></b> + +echo <font color="#FF0000">"Failover of zone $zone to $MASTER completed"</font> +<b><font color="#0000FF">return</font></b> <font color="#993399">1</font> +</pre> +<br /> +<span>A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output.</span><br /> +<br /> +<span>The nameserver is running on both VMs, and both are configured to be "master" DNS servers so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case.</span><br /> +<br /> +<span>Check out the whole script here:</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh'>https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh</a><br /> +<br /> +<h3 style='display: inline'>Fairly cheap and geo-redundant</h3><br /> +<br /> +<span>I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany).</span><br /> +<br /> +<a class='textlink' href='https://openbsd.amsterdam'>https://openbsd.amsterdam</a><br /> +<a class='textlink' href='https://www.hetzner.cloud'>https://www.hetzner.cloud</a><br /> +<br /> +<span>I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required.</span><br /> +<br /> +<span>A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions.</span><br /> +<br /> +<h3 style='display: inline'>Failover time and split-brain</h3><br /> +<br /> +<span>A DNS failover doesn't happen immediately. I've configured a DNS TTL of <span class='inlinecode'>300</span> seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover).</span><br /> +<br /> +<span>A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated.</span><br /> +<br /> +<h3 style='display: inline'>Failover support for multiple protocols</h3><br /> +<br /> +<span>With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's <span class='inlinecode'>httpd</span> (OpenBSD's HTTP server) and <span class='inlinecode'>relayd</span> (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entry, which hosts receive the requests.</span><br /> +<br /> +<a class='textlink' href='https://man.openbsd.org/httpd.8'>https://man.openbsd.org/httpd.8</a><br /> +<a class='textlink' href='https://man.openbsd.org/relayd.8'>https://man.openbsd.org/relayd.8</a><br /> +<br /> +<span>For example, the master is responsible for the <span class='inlinecode'>https://www.foo.zone</span> and <span class='inlinecode'>https://foo.zone</span> hosts, whereas the standby can be reached via <span class='inlinecode'>https://standby.foo.zone</span> (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. <span class='inlinecode'>irregular.ninja</span>, <span class='inlinecode'>paul.buetow.org</span> and so on. The same applies to my Gemini capsules for <span class='inlinecode'>gemini://foo.zone</span>, <span class='inlinecode'>gemini://standby.foo.zone</span>, <span class='inlinecode'>gemini://paul.buetow.org</span> and <span class='inlinecode'>gemini://standby.paul.buetow.org</span>.</span><br /> +<br /> +<span>On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)!</span><br /> +<br /> +<h3 style='display: inline'>Let's encrypt TLS certificates</h3><br /> +<br /> +<span>All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate.</span><br /> +<br /> +<span>If the master always serves <span class='inlinecode'>foo.zone</span> and the standby always <span class='inlinecode'>standby.foo.zone</span>, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for <span class='inlinecode'>foo.zone</span> and the new standby wouldn't have a valid certificate for <span class='inlinecode'>standby.foo.zone</span> which would lead to TLS errors on the clients.</span><br /> +<br /> +<span>As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that:</span><br /> +<br /> +<ul> +<li>In an odd week number, the first server is the default master</li> +<li>In an even week number, the second server is the default master.</li> +</ul><br /> +<span>Which translates to:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900"># Weekly auto-failover for Let's Encrypt automation</font></i> +<b><font color="#0000FF">local</font></b> -i -r <font color="#009900">week_of_the_year</font><font color="#990000">=</font><font color="#009900">$(</font>date <font color="#990000">+%</font>U<font color="#990000">)</font> +<b><font color="#0000FF">if</font></b> <font color="#990000">[</font> <font color="#009900">$(</font><font color="#990000">(</font> week_of_the_year <font color="#990000">%</font> <font color="#993399">2</font> <font color="#990000">))</font> -eq <font color="#993399">0</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + <b><font color="#0000FF">local</font></b> <font color="#009900">tmp</font><font color="#990000">=</font><font color="#009900">$master</font> + <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$standby</font> + <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$tmp</font> +<b><font color="#0000FF">fi</font></b> +</pre> +<br /> +<span>This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM.</span><br /> +<br /> +<span>The ACME automation is yet another daily CRON script <span class='inlinecode'>/usr/local/bin/acme.sh</span>. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt. </span><br /> +<br /> +<span>Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty.</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/acme.sh.tpl'><span class='inlinecode'>acme.sh.tpl</span> - Rex template for the <span class='inlinecode'>acme.sh</span> script of mine.</a><br /> +<a class='textlink' href='https://man.openbsd.org/acme-client.1'>https://man.openbsd.org/acme-client.1</a><br /> +<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's Encrypt with OpenBSD and Rex</a><br /> +<br /> +<h3 style='display: inline'>Monitoring</h3><br /> +<br /> +<span>CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed. </span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/gogios'>https://codeberg.org/snonux/gogios</a><br /> +<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>KISS server monitoring with Gogios</a><br /> +<span> </span><br /> +<h3 style='display: inline'>Rex automation</h3><br /> +<br /> +<span>I use Rexify, a friendly configuration management system that allows automatic deployment and configuration.</span><br /> +<br /> +<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends'>https://codeberg.org/snonux/rexfiles/src/branch/master/frontends</a><br /> +<br /> +<h2 style='display: inline'>More HA</h2><br /> +<br /> +<span>Other high-available services running on my OpenBSD VMs are my MTAs for mail forwarding (OpenSMTPD) and the authoritative DNS servers (<span class='inlinecode'>nsd</span>) for all my domains. No particular HA setup is required, though, as the protocols (SMTP and DNS) already take care of the failover to the next available host! </span><br /> +<br /> +<span>As a password manager, I use <span class='inlinecode'>geheim</span>, a command-line tool I wrote in Ruby with encrypted files in a git repository (I even have it installed in Termux on my Phone). For HA reasons, I simply updated the client code so that it always synchronises the database with both servers when I run the <span class='inlinecode'>sync</span> command there. </span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/geheim'>https://codeberg.org/snonux/geheim</a><br /> +<br /> +<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> +<br /> +<span>Other *BSD and KISS related posts are:</span><br /> +<br /> +<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br /> +<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br /> +<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br /> +<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios</a><br /> +<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br /> +<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br /> +<br /> +<a class='textlink' href='../'>Back to the main site</a><br /> +<p class="footer"> +Generated by <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 2.1.0-release</a> | +served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | +<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a> +</p> +</body> +</html> diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index e0135d68..e8a316bf 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,12 +1,332 @@ <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> - <updated>2024-03-03T11:31:05+02:00</updated> + <updated>2024-03-30T22:13:12+02:00</updated> <title>foo.zone feed</title> <subtitle>To be in the .zone!</subtitle> <link href="https://foo.zone/gemfeed/atom.xml" rel="self" /> <link href="https://foo.zone/" /> <id>https://foo.zone/</id> <entry> + <title>KISS high-availability with OpenBSD</title> + <link href="https://foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html" /> + <id>https://foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html</id> + <updated>2024-03-30T22:12:56+02:00</updated> + <author> + <name>Paul Buetow aka snonux</name> + <email>paul@dev.buetow.org</email> + </author> + <summary>Art by Michael J. Penick (mod. by Paul B)</summary> + <content type="xhtml"> + <div xmlns="http://www.w3.org/1999/xhtml"> + <h1 style='display: inline'>KISS high-availability with OpenBSD</h1><br /> +<br /> +<span class='quote'>Published at 2024-03-30T22:12:56+02:00</span><br /> +<br /> +<pre> +Art by Michael J. Penick (mod. by Paul B) + + __________ + / nsd tower\ ( + /____________\ (\) awk-ward + |:_:_:_:_:_| )) plant + |_:_,--.:_:| dig-bubble (\// ) + |:_:|__|_:_| relayd-castle _ ) )) (( + _ |_ _ :_:| _ _ _ (_) (((( /)\` + | |_| |_| | _| | |_| |_| | o \\)) (( ( + \_:_:_:_:/|_|_|_|\:_:_:_:_/ . (( )))) + |_,-._:_:_:_:_:_:_:_.-,_| )) ((// + |:|_|:_:_:,---,:_:_:|_|:| ,-. )/ + |_:_:_:_,'puffy `,_:_:_:_| _ o ,;'))(( + |:_:_:_/ _ | _ \_:_:_:| (_O (( )) +_____|_:_:_| (o)-(o) |_:_:_|--'`-. ,--. ksh under-water (((\'/ + ', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\ /,---.~ goat \`)) +. ` |_:_:_| \`-'/ |_:_:_|. ` . ` /()\.__( ) .,-----'`-\(( sed-root + ', ;|:_:_:| `-' |:_:_:| ', ; ', ; `--'| \ ', ; ', ; ',')).,-- +. ` MJP ` . ` . ` . ` . httpd-soil ` . . ` . ` . ` . ` . ` + ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; + +</pre> +<br /> +<span>I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like BGP, LVS/IPVS, ldirectord, Pacemaker, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work. </span><br /> +<br /> +<span>But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space—something as KISS (keep it simple and stupid) as possible.</span><br /> +<br /> +<span>It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway.</span><br /> +<br /> +<span class='quote'>PS: ASCII-art reflects the OpenBSD under-water world with all the tools available in the base system.</span><br /> +<br /> +<h2 style='display: inline'>My auto-failover requirements</h2><br /> +<br /> +<ul> +<li>Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible. </li> +<li>Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month).</li> +<li>It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.</li> +<li>It should be geo-redundant. </li> +<li>It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.</li> +<li>Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.</li> +<li>Let's Encrypt TLS certificates should always work (before and after a failover).</li> +<li>Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover.</li> +<li>Don't configure everything manually. The configuration should be automated and reproducible.</li> +</ul><br /> +<h2 style='display: inline'>My HA solution</h2><br /> +<br /> +<h3 style='display: inline'>Only OpenBSD base installation required</h3><br /> +<br /> +<span>My HA solution for Web and Gemini is based on DNS (OpenBSD's <span class='inlinecode'>nsd</span>) and a simple shell script (OpenBSD's <span class='inlinecode'>ksh</span> and some little <span class='inlinecode'>sed</span> and <span class='inlinecode'>awk</span> and <span class='inlinecode'>grep</span>). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system.</span><br /> +<br /> +<a class='textlink' href='https://man.OpenBSD.org/nsd.8'>https://man.OpenBSD.org/nsd.8</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/ksh'>https://man.OpenBSD.org/ksh</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/awk'>https://man.OpenBSD.org/awk</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/sed'>https://man.OpenBSD.org/sed</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/dig'>https://man.OpenBSD.org/dig</a><br /> +<a class='textlink' href='https://man.OpenBSD.org/ftp'>https://man.OpenBSD.org/ftp</a><br /> +<br /> +<span>I also used the <span class='inlinecode'>dig</span> (for DNS checks) and <span class='inlinecode'>ftp</span> (for HTTP/HTTPS checks) programs. </span><br /> +<br /> +<span>The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The <span class='inlinecode'>ksh</span> script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM: </span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900">#!/bin/ksh</font></i> + +<font color="#009900">ZONES_DIR</font><font color="#990000">=</font>/var/nsd/zones/master<font color="#990000">/</font> +<font color="#009900">DEFAULT_MASTER</font><font color="#990000">=</font>fishfinger<font color="#990000">.</font>buetow<font color="#990000">.</font>org +<font color="#009900">DEFAULT_STANDBY</font><font color="#990000">=</font>blowfish<font color="#990000">.</font>buetow<font color="#990000">.</font>org + +<b><font color="#000000">determine_master_and_standby ()</font></b> { + <b><font color="#0000FF">local</font></b> <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$DEFAULT_MASTER</font> + <b><font color="#0000FF">local</font></b> <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$DEFAULT_STANDBY</font> + + <font color="#990000">.</font> + <font color="#990000">.</font> + <font color="#990000">.</font> + + <b><font color="#0000FF">local</font></b> -i <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">1</font> + <b><font color="#0000FF">if</font></b> <font color="#990000">!</font> ftp -<font color="#993399">4</font> -o - https<font color="#990000">://</font><font color="#009900">$master</font>/index<font color="#990000">.</font>txt <font color="#990000">|</font> grep -q <font color="#FF0000">"Welcome to $master"</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"https://$master/index.txt IPv4 health check failed"</font> + <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">0</font> + <b><font color="#0000FF">elif</font></b> <font color="#990000">!</font> ftp -<font color="#993399">6</font> -o - https<font color="#990000">://</font><font color="#009900">$master</font>/index<font color="#990000">.</font>txt <font color="#990000">|</font> grep -q <font color="#FF0000">"Welcome to $master"</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"https://$master/index.txt IPv6 health check failed"</font> + <font color="#009900">health_ok</font><font color="#990000">=</font><font color="#993399">0</font> + <b><font color="#0000FF">fi</font></b> + <b><font color="#0000FF">if</font></b> <font color="#990000">[</font> <font color="#009900">$health_ok</font> -eq <font color="#993399">0</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + <b><font color="#0000FF">local</font></b> <font color="#009900">tmp</font><font color="#990000">=</font><font color="#009900">$master</font> + <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$standby</font> + <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$tmp</font> + <b><font color="#0000FF">fi</font></b> + + <font color="#990000">.</font> + <font color="#990000">.</font> + <font color="#990000">.</font> +} +</pre> +<br /> +<span>The failover scripts looks for the <span class='inlinecode'> ; Enable failover</span> string in the DNS zone files and swaps the <span class='inlinecode'>A</span> and <span class='inlinecode'>AAAA</span> records of the DNS entries accordingly:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>fishfinger$ grep failover /var/nsd/zones/master/foo<font color="#990000">.</font>zone<font color="#990000">.</font>zone + <font color="#993399">300</font> IN A <font color="#993399">46.23</font><font color="#990000">.</font><font color="#993399">94.99</font> <font color="#990000">;</font> Enable failover + <font color="#993399">300</font> IN AAAA 2a03<font color="#990000">:</font><font color="#993399">6000</font><font color="#990000">:</font>6f67<font color="#990000">:</font><font color="#993399">624</font><font color="#990000">::</font><font color="#993399">99</font> <font color="#990000">;</font> Enable failover +www <font color="#993399">300</font> IN A <font color="#993399">46.23</font><font color="#990000">.</font><font color="#993399">94.99</font> <font color="#990000">;</font> Enable failover +www <font color="#993399">300</font> IN AAAA 2a03<font color="#990000">:</font><font color="#993399">6000</font><font color="#990000">:</font>6f67<font color="#990000">:</font><font color="#993399">624</font><font color="#990000">::</font><font color="#993399">99</font> <font color="#990000">;</font> Enable failover +standby <font color="#993399">300</font> IN A <font color="#993399">23.88</font><font color="#990000">.</font><font color="#993399">35.144</font> <font color="#990000">;</font> Enable failover +standby <font color="#993399">300</font> IN AAAA 2a01<font color="#990000">:</font>4f8<font color="#990000">:</font>c17<font color="#990000">:</font>20f1<font color="#990000">::</font><font color="#993399">42</font> <font color="#990000">;</font> Enable failover +</pre> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><b><font color="#000000">tramsform ()</font></b> { + sed -E <font color="#FF0000">'</font> +<font color="#FF0000"> /IN A .*; Enable failover/ {</font> +<font color="#FF0000"> /^standby/! {</font> +<font color="#FF0000"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN A '</font><font color="#009900">$(</font>cat /var/nsd/run/master_a<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /^standby/ {</font> +<font color="#FF0000"> s/^(.*) 300 IN A (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN A '</font><font color="#009900">$(</font>cat /var/nsd/run/standby_a<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /IN AAAA .*; Enable failover/ {</font> +<font color="#FF0000"> /^standby/! {</font> +<font color="#FF0000"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN AAAA '</font><font color="#009900">$(</font>cat /var/nsd/run/master_aaaa<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> /^standby/ {</font> +<font color="#FF0000"> s/^(.*) 300 IN AAAA (.*) ; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> 300 IN AAAA '</font><font color="#009900">$(</font>cat /var/nsd/run/standby_aaaa<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> / ; serial/ {</font> +<font color="#FF0000"> s/^( +) ([0-9]+) .*; (.*)/</font><font color="#CC33CC">\1</font><font color="#FF0000"> '</font><font color="#009900">$(</font>date <font color="#990000">+%</font>s<font color="#990000">)</font><font color="#FF0000">' ; </font><font color="#CC33CC">\3</font><font color="#FF0000">/;</font> +<font color="#FF0000"> }</font> +<font color="#FF0000"> '</font> +} +</pre> +<br /> +<span>After the failover, the script reloads <span class='inlinecode'>nsd</span> and performs a sanity check to see if DNS still works. If not, a rollback will be performed:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900"># Race condition (e.g. script execution abored in the middle of the previous run)</font></i> +<b><font color="#0000FF">if</font></b> <font color="#990000">[</font> -f <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + mv <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#009900">$zone_file</font> +<b><font color="#0000FF">fi</font></b> + +cat <font color="#009900">$zone_file</font> <font color="#990000">|</font> transform <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp + +grep -v <font color="#FF0000">' ; serial'</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp +grep -v <font color="#FF0000">' ; serial'</font> <font color="#009900">$zone_file</font> <font color="#990000">></font> <font color="#009900">$zone_file</font><font color="#990000">.</font>old<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp + +echo <font color="#FF0000">"Has zone $zone_file changed?"</font> +<b><font color="#0000FF">if</font></b> diff -u <font color="#009900">$zone_file</font><font color="#990000">.</font>old<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>noserial<font color="#990000">.</font>tmp<font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"The zone $zone_file hasn't changed"</font> + rm <font color="#009900">$zone_file</font><font color="#990000">.*.</font>tmp + <b><font color="#0000FF">return</font></b> <font color="#993399">0</font> +<b><font color="#0000FF">fi</font></b> + +cp <font color="#009900">$zone_file</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>bak +mv <font color="#009900">$zone_file</font><font color="#990000">.</font>new<font color="#990000">.</font>tmp <font color="#009900">$zone_file</font> +rm <font color="#009900">$zone_file</font><font color="#990000">.*.</font>tmp +echo <font color="#FF0000">"Reloading nsd"</font> +nsd-control reload + +<b><font color="#0000FF">if</font></b> <font color="#990000">!</font> zone_is_ok <font color="#009900">$zone</font><font color="#990000">;</font> <b><font color="#0000FF">then</font></b> + echo <font color="#FF0000">"Rolling back $zone_file changes"</font> + cp <font color="#009900">$zone_file</font> <font color="#009900">$zone_file</font><font color="#990000">.</font>invalid + mv <font color="#009900">$zone_file</font><font color="#990000">.</font>bak <font color="#009900">$zone_file</font> + echo <font color="#FF0000">"Reloading nsd"</font> + nsd-control reload + zone_is_ok <font color="#009900">$zone</font> + <b><font color="#0000FF">return</font></b> <font color="#993399">3</font> +<b><font color="#0000FF">fi</font></b> + +<b><font color="#0000FF">for</font></b> cleanup <b><font color="#0000FF">in</font></b> invalid bak<font color="#990000">;</font> <b><font color="#0000FF">do</font></b> + <b><font color="#0000FF">if</font></b> <font color="#990000">[</font> -f <font color="#009900">$zone_file</font><font color="#990000">.</font><font color="#009900">$cleanup</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + rm <font color="#009900">$zone_file</font><font color="#990000">.</font><font color="#009900">$cleanup</font> + <b><font color="#0000FF">fi</font></b> +<b><font color="#0000FF">done</font></b> + +echo <font color="#FF0000">"Failover of zone $zone to $MASTER completed"</font> +<b><font color="#0000FF">return</font></b> <font color="#993399">1</font> +</pre> +<br /> +<span>A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output.</span><br /> +<br /> +<span>The nameserver is running on both VMs, and both are configured to be "master" DNS servers so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case.</span><br /> +<br /> +<span>Check out the whole script here:</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh'>https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh</a><br /> +<br /> +<h3 style='display: inline'>Fairly cheap and geo-redundant</h3><br /> +<br /> +<span>I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany).</span><br /> +<br /> +<a class='textlink' href='https://openbsd.amsterdam'>https://openbsd.amsterdam</a><br /> +<a class='textlink' href='https://www.hetzner.cloud'>https://www.hetzner.cloud</a><br /> +<br /> +<span>I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required.</span><br /> +<br /> +<span>A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions.</span><br /> +<br /> +<h3 style='display: inline'>Failover time and split-brain</h3><br /> +<br /> +<span>A DNS failover doesn't happen immediately. I've configured a DNS TTL of <span class='inlinecode'>300</span> seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover).</span><br /> +<br /> +<span>A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated.</span><br /> +<br /> +<h3 style='display: inline'>Failover support for multiple protocols</h3><br /> +<br /> +<span>With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's <span class='inlinecode'>httpd</span> (OpenBSD's HTTP server) and <span class='inlinecode'>relayd</span> (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entry, which hosts receive the requests.</span><br /> +<br /> +<a class='textlink' href='https://man.openbsd.org/httpd.8'>https://man.openbsd.org/httpd.8</a><br /> +<a class='textlink' href='https://man.openbsd.org/relayd.8'>https://man.openbsd.org/relayd.8</a><br /> +<br /> +<span>For example, the master is responsible for the <span class='inlinecode'>https://www.foo.zone</span> and <span class='inlinecode'>https://foo.zone</span> hosts, whereas the standby can be reached via <span class='inlinecode'>https://standby.foo.zone</span> (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. <span class='inlinecode'>irregular.ninja</span>, <span class='inlinecode'>paul.buetow.org</span> and so on. The same applies to my Gemini capsules for <span class='inlinecode'>https://foo.zone</span>, <span class='inlinecode'>https://standby.foo.zone</span>, <span class='inlinecode'>https://paul.buetow.org</span> and <span class='inlinecode'>https://standby.paul.buetow.org</span>.</span><br /> +<br /> +<span>On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)!</span><br /> +<br /> +<h3 style='display: inline'>Let's encrypt TLS certificates</h3><br /> +<br /> +<span>All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate.</span><br /> +<br /> +<span>If the master always serves <span class='inlinecode'>foo.zone</span> and the standby always <span class='inlinecode'>standby.foo.zone</span>, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for <span class='inlinecode'>foo.zone</span> and the new standby wouldn't have a valid certificate for <span class='inlinecode'>standby.foo.zone</span> which would lead to TLS errors on the clients.</span><br /> +<br /> +<span>As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that:</span><br /> +<br /> +<ul> +<li>In an odd week number, the first server is the default master</li> +<li>In an even week number, the second server is the default master.</li> +</ul><br /> +<span>Which translates to:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="#9A1900"># Weekly auto-failover for Let's Encrypt automation</font></i> +<b><font color="#0000FF">local</font></b> -i -r <font color="#009900">week_of_the_year</font><font color="#990000">=</font><font color="#009900">$(</font>date <font color="#990000">+%</font>U<font color="#990000">)</font> +<b><font color="#0000FF">if</font></b> <font color="#990000">[</font> <font color="#009900">$(</font><font color="#990000">(</font> week_of_the_year <font color="#990000">%</font> <font color="#993399">2</font> <font color="#990000">))</font> -eq <font color="#993399">0</font> <font color="#990000">];</font> <b><font color="#0000FF">then</font></b> + <b><font color="#0000FF">local</font></b> <font color="#009900">tmp</font><font color="#990000">=</font><font color="#009900">$master</font> + <font color="#009900">master</font><font color="#990000">=</font><font color="#009900">$standby</font> + <font color="#009900">standby</font><font color="#990000">=</font><font color="#009900">$tmp</font> +<b><font color="#0000FF">fi</font></b> +</pre> +<br /> +<span>This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM.</span><br /> +<br /> +<span>The ACME automation is yet another daily CRON script <span class='inlinecode'>/usr/local/bin/acme.sh</span>. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt. </span><br /> +<br /> +<span>Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty.</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/acme.sh.tpl'><span class='inlinecode'>acme.sh.tpl</span> - Rex template for the <span class='inlinecode'>acme.sh</span> script of mine.</a><br /> +<a class='textlink' href='https://man.openbsd.org/acme-client.1'>https://man.openbsd.org/acme-client.1</a><br /> +<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>Let's Encrypt with OpenBSD and Rex</a><br /> +<br /> +<h3 style='display: inline'>Monitoring</h3><br /> +<br /> +<span>CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed. </span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/gogios'>https://codeberg.org/snonux/gogios</a><br /> +<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>KISS server monitoring with Gogios</a><br /> +<span> </span><br /> +<h3 style='display: inline'>Rex automation</h3><br /> +<br /> +<span>I use Rexify, a friendly configuration management system that allows automatic deployment and configuration.</span><br /> +<br /> +<a class='textlink' href='https://www.rexify.org'>https://www.rexify.org</a><br /> +<a class='textlink' href='https://codeberg.org/snonux/rexfiles/src/branch/master/frontends'>https://codeberg.org/snonux/rexfiles/src/branch/master/frontends</a><br /> +<br /> +<h2 style='display: inline'>More HA</h2><br /> +<br /> +<span>Other high-available services running on my OpenBSD VMs are my MTAs for mail forwarding (OpenSMTPD) and the authoritative DNS servers (<span class='inlinecode'>nsd</span>) for all my domains. No particular HA setup is required, though, as the protocols (SMTP and DNS) already take care of the failover to the next available host! </span><br /> +<br /> +<span>As a password manager, I use <span class='inlinecode'>geheim</span>, a command-line tool I wrote in Ruby with encrypted files in a git repository (I even have it installed in Termux on my Phone). For HA reasons, I simply updated the client code so that it always synchronises the database with both servers when I run the <span class='inlinecode'>sync</span> command there. </span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/geheim'>https://codeberg.org/snonux/geheim</a><br /> +<br /> +<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> +<br /> +<span>Other *BSD and KISS related posts are:</span><br /> +<br /> +<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br /> +<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br /> +<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br /> +<a class='textlink' href='./2023-06-01-kiss-server-monitoring-with-gogios.html'>2023-06-01 KISS server monitoring with Gogios</a><br /> +<a class='textlink' href='./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.html'>2023-10-29 KISS static web photo albums with <span class='inlinecode'>photoalbum.sh</span></a><br /> +<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br /> +<br /> +<a class='textlink' href='../'>Back to the main site</a><br /> + </div> + </content> + </entry> + <entry> <title>A fine Fyne Android app for quickly logging ideas programmed in Go</title> <link href="https://foo.zone/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.html" /> <id>https://foo.zone/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.html</id> @@ -8397,479 +8717,4 @@ assert<font color="#990000">::</font>equals <font color="#FF0000">"$(generate::m </div> </content> </entry> - <entry> - <title>Personal Bash coding style guide</title> - <link href="https://foo.zone/gemfeed/2021-05-16-personal-bash-coding-style-guide.html" /> - <id>https://foo.zone/gemfeed/2021-05-16-personal-bash-coding-style-guide.html</id> - <updated>2021-05-16T14:51:57+01:00</updated> - <author> - <name>Paul Buetow aka snonux</name> - <email>paul@dev.buetow.org</email> - </author> - <summary>Lately, I have been polishing and writing a lot of Bash code. Not that I never wrote a lot of Bash, but now as I also looked through the Google Shell Style Guide, I thought it is time also to write my thoughts on that. I agree with that guide in most, but not in all points. </summary> - <content type="xhtml"> - <div xmlns="http://www.w3.org/1999/xhtml"> - <h1 style='display: inline'>Personal Bash coding style guide</h1><br /> -<br /> -<span class='quote'>Published at 2021-05-16T14:51:57+01:00</span><br /> -<br /> -<pre> - .---------------------------. - /,--..---..---..---..---..--. `. - //___||___||___||___||___||___\_| - [j__ ######################## [_| - \============================| - .==| |"""||"""||"""||"""| |"""|| -/======"---""---""---""---"=| =|| -|____ []* ____ | ==|| -// \\ // \\ |===|| hjw -"\__/"---------------"\__/"-+---+' -</pre> -<br /> -<span>Lately, I have been polishing and writing a lot of Bash code. Not that I never wrote a lot of Bash, but now as I also looked through the Google Shell Style Guide, I thought it is time also to write my thoughts on that. I agree with that guide in most, but not in all points. </span><br /> -<br /> -<a class='textlink' href='https://google.github.io/styleguide/shellguide.html'>Google Shell Style Guide</a><br /> -<br /> -<h2 style='display: inline'>My modifications</h2><br /> -<br /> -<span>These are my modifications to the Google Guide.</span><br /> -<br /> -<h3 style='display: inline'>Shebang</h3><br /> -<br /> -<span>Google recommends using always...</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><i><font color="#9A1900">#!/bin/bash </font></i> -</pre> -<br /> -<span>... as the shebang line, but that does not work on all Unix and Unix-like operating systems (e.g., the *BSDs don't have Bash installed to /bin/bash). Better is:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><i><font color="#9A1900">#!/usr/bin/env bash</font></i> -</pre> -<br /> -<h3 style='display: inline'>Two space soft-tabs indentation</h3><br /> -<br /> -<span>I know there have been many tab- and soft-tab wars on this planet. Google recommends using two space soft-tabs for Bash scripts. </span><br /> -<br /> -<span>I don't care if I use two or four space indentations. I agree, however, that we should not use tabs. I tend to use four-space soft-tabs as that's how I currently configured Vim for any programming language. What matters most, though, is consistency within the same script/project.</span><br /> -<br /> -<span>Google also recommends limiting the line length to 80 characters. For some people, that seems to be an old habit from the '80s, where all computer terminals couldn't display longer lines. But I think that the 80 character mark is still a good practice, at least for shell scripts. For example, I am often writing code on a Microsoft Go Tablet PC (running Linux, of course), and it comes in convenient if the lines are not too long due to the relatively small display on the device.</span><br /> -<br /> -<span>I hit the 80 character line length quicker with the four spaces than with two spaces, but that makes me refactor the Bash code more aggressively, which is a good thing. </span><br /> -<br /> -<h3 style='display: inline'>Breaking long pipes</h3><br /> -<br /> -<span>Google recommends breaking up long pipes like this:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><i><font color="#9A1900"># All fits on one line</font></i> -command1 <font color="#990000">|</font> command2 - -<i><font color="#9A1900"># Long commands</font></i> -command1 <font color="#990000">\</font> - <font color="#990000">|</font> command2 <font color="#990000">\</font> - <font color="#990000">|</font> command3 <font color="#990000">\</font> - <font color="#990000">|</font> command4 -</pre> -<br /> -<span>I think there is a better way like the following, which is less noisy. The pipe | already indicates the Bash that another command is expected, thus making the explicit line breaks with \ obsolete:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><i><font color="#9A1900"># Long commands</font></i> -command1 <font color="#990000">|</font> - command2 <font color="#990000">|</font> - command3 <font color="#990000">|</font> - command4 -</pre> -<br /> -<span class='quote'>Update: It's 2023 now, and I have changed my mind. I think Google's way is the better one. It may be a bit more to type, but the leading <span class='inlinecode'>|</span> are a nice eye catcher, so you know immediately what is going on!</span><br /> -<br /> -<h3 style='display: inline'>Quoting your variables</h3><br /> -<br /> -<span>Google recommends always quote your variables. Generally, it would be best if you did that only for variables where you are unsure about the content/values of the variables (e.g., content is from an external input source and may contain whitespace or other special characters). In my opinion, the code will become quite noisy when you always quote your variables like this:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#000000">greet ()</font></b> { - <b><font color="#0000FF">local</font></b> -r <font color="#009900">greeting</font><font color="#990000">=</font><font color="#FF0000">"${1}"</font> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">name</font><font color="#990000">=</font><font color="#FF0000">"${2}"</font> - echo <font color="#FF0000">"${greeting} ${name}!"</font> -} -</pre> -<br /> -<span>In this particular example, I agree that you should quote them as you don't know the input (are there, for example, whitespace characters?). But if you are sure that you are only using simple bare words, then I think that the code looks much cleaner when you do this instead:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#000000">say_hello_to_paul ()</font></b> { - <b><font color="#0000FF">local</font></b> -r <font color="#009900">greeting</font><font color="#990000">=</font>Hello - <b><font color="#0000FF">local</font></b> -r <font color="#009900">name</font><font color="#990000">=</font>Paul - echo <font color="#FF0000">"$greeting $name!"</font> -} -</pre> -<br /> -<span>You see, I also omitted the curly braces { } around the variables. I only use the curly braces around variables when it makes the code either easier/clearer to read or if it is necessary to use them:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#0000FF">declare</font></b> <font color="#009900">FOO</font><font color="#990000">=</font>bar -<i><font color="#9A1900"># Curly braces around FOO are necessary</font></i> -echo <font color="#FF0000">"foo${FOO}baz"</font> -</pre> -<br /> -<span>A few more words on always quoting the variables: For the sake of consistency (and for making ShellCheck happy), I am not against quoting everything I encounter. I also think that the larger the Bash script becomes, the more critical it becomes always to quote variables. That's because it will be more likely that you might not remember that some of the functions don't work on values with spaces in them, for example. It's just that I won't quote everything in every small script I write. </span><br /> -<br /> -<h3 style='display: inline'>Prefer built-in commands over external commands</h3><br /> -<br /> -<span>Google recommends using the built-in commands over available external commands where possible:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><i><font color="#9A1900"># Prefer this:</font></i> -<font color="#009900">addition</font><font color="#990000">=</font><font color="#009900">$(</font><font color="#990000">(</font> X <font color="#990000">+</font> Y <font color="#990000">))</font> -<font color="#009900">substitution</font><font color="#990000">=</font><font color="#FF0000">"${string/#foo/bar}"</font> - -<i><font color="#9A1900"># Instead of this:</font></i> -<font color="#009900">addition</font><font color="#990000">=</font><font color="#FF0000">"$(expr "</font><font color="#009900">${X}</font><font color="#FF0000">" + "</font><font color="#009900">${Y}</font><font color="#FF0000">")"</font> -<font color="#009900">substitution</font><font color="#990000">=</font><font color="#FF0000">"$(echo "</font><font color="#009900">${string}</font><font color="#FF0000">" | sed -e 's/^foo/bar/')"</font> -</pre> -<br /> -<span>I can't entirely agree here. The external commands (especially sed) are much more sophisticated and powerful than the built-in Bash versions. Sed can do much more than the Bash can ever do by itself when it comes to text manipulation (the name "sed" stands for streaming editor, after all).</span><br /> -<br /> -<span>I prefer to do light text processing with the Bash built-ins and more complicated text processing with external programs such as sed, grep, awk, cut, and tr. However, there is also medium-light text processing where I would want to use external programs. That is so because I remember using them better than the Bash built-ins. The Bash can get relatively obscure here (even Perl will be more readable then - Side note: I love Perl).</span><br /> -<br /> -<span>Also, you would like to use an external command for floating-point calculation (e.g., bc) instead of using the Bash built-ins (worth noticing that ZSH supports built-in floating-points).</span><br /> -<br /> -<span>I even didn't get started with what you can do with awk (especially GNU Awk), a fully-fledged programming language. Tiny Awk snippets tend to be used quite often in Shell scripts without honouring the real power of Awk. But if you did everything in Perl or Awk or another scripting language, then it wouldn't be a Bash script anymore, wouldn't it? ;-)</span><br /> -<br /> -<h2 style='display: inline'>My additions</h2><br /> -<br /> -<h3 style='display: inline'>Use of 'yes' and 'no'</h3><br /> -<br /> -<span>Bash does not support a boolean type. I tend just to use the strings 'yes' and 'no' here. I used 0 for false and 1 for true for some time, but I think that the yes/no strings are easier to read. Yes, the Bash script would need to perform string comparisons on every check, but if performance is crucial to you, you wouldn't want to use a Bash script anyway, correct?</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#0000FF">declare</font></b> -r <font color="#009900">SUGAR_FREE</font><font color="#990000">=</font>yes -<b><font color="#0000FF">declare</font></b> -r <font color="#009900">I_NEED_THE_BUZZ</font><font color="#990000">=</font>no - -<b><font color="#000000">buy_soda ()</font></b> { - <b><font color="#0000FF">local</font></b> -r <font color="#009900">sugar_free</font><font color="#990000">=</font><font color="#009900">$1</font> - - <b><font color="#0000FF">if</font></b> <font color="#990000">[[</font> <font color="#009900">$sugar_free</font> <font color="#990000">==</font> yes <font color="#990000">]];</font> <b><font color="#0000FF">then</font></b> - echo <font color="#FF0000">'Diet Dr. Pepper'</font> - <b><font color="#0000FF">else</font></b> - echo <font color="#FF0000">'Pepsi Coke'</font> - <b><font color="#0000FF">fi</font></b> -} - -buy_soda <font color="#009900">$I_NEED_THE_BUZZ</font> -</pre> -<br /> -<h3 style='display: inline'>Non-evil alternative to variable assignments via eval</h3><br /> -<br /> -<span>Google is in the opinion that eval should be avoided. I think so too. They list these examples in their guide:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><i><font color="#9A1900"># What does this set?</font></i> -<i><font color="#9A1900"># Did it succeed? In part or whole?</font></i> -<b><font color="#0000FF">eval</font></b> <font color="#009900">$(set_my_variables)</font> - -<i><font color="#9A1900"># What happens if one of the returned values has a space in it?</font></i> -<font color="#009900">variable</font><font color="#990000">=</font><font color="#FF0000">"$(eval some_function)"</font> -</pre> -<br /> -<span>However, if I want to read variables from another file, I don't have to use eval here. I only have to source the file:</span><br /> -<br /> -<pre> -% cat vars.source.sh -declare foo=bar -declare bar=baz -declare bay=foo - -% bash -c 'source vars.source.sh; echo $foo $bar $baz' -bar baz foo -</pre> -<br /> -<span>And suppose I want to assign variables dynamically. In that case, I could just run an external script and source its output (This is how you could do metaprogramming in Bash without the use of eval - write code which produces code for immediate execution):</span><br /> -<br /> -<pre> -% cat vars.sh -#!/usr/bin/env bash -cat <<END -declare date="$(date)" -declare user=$USER -END - -% bash -c 'source <(./vars.sh); echo "Hello $user, it is $date"' -Hello paul, it is Sat 15 May 19:21:12 BST 2021 -</pre> -<br /> -<span>The downside is that ShellCheck won't be able to follow the dynamic sourcing anymore.</span><br /> -<br /> -<h3 style='display: inline'>Prefer pipes over arrays for list processing</h3><br /> -<br /> -<span>When I do list processing in Bash, I prefer to use pipes. You can chain them through Bash functions as well, which is pretty neat. Usually, my list processing scripts are of a structure like this:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#000000">filter_lines ()</font></b> { - echo <font color="#FF0000">'Start filtering lines in a fancy way!'</font> <font color="#990000">>&</font><font color="#993399">2</font> - grep <font color="#990000">...</font> <font color="#990000">|</font> sed <font color="#990000">....</font> -} - -<b><font color="#000000">process_lines ()</font></b> { - echo <font color="#FF0000">'Start processing line by line!'</font> <font color="#990000">>&</font><font color="#993399">2</font> - <b><font color="#0000FF">while</font></b> <b><font color="#0000FF">read</font></b> -r line<font color="#990000">;</font> <b><font color="#0000FF">do</font></b> - <font color="#990000">...</font> <b><font color="#0000FF">do</font></b> something and produce a result<font color="#990000">...</font> - echo <font color="#FF0000">"$result"</font> - <b><font color="#0000FF">done</font></b> -} - -<i><font color="#9A1900"># Do some post-processing of the data</font></i> -<b><font color="#000000">postprocess_lines ()</font></b> { - echo <font color="#FF0000">'Start removing duplicates!'</font> <font color="#990000">>&</font><font color="#993399">2</font> - sort -u -} - -<b><font color="#000000">genreate_report ()</font></b> { - echo <font color="#FF0000">'My boss wants to have a report!'</font> <font color="#990000">>&</font><font color="#993399">2</font> - tee outfile<font color="#990000">.</font>txt - wc -l outfile<font color="#990000">.</font>txt -} - -<b><font color="#000000">main ()</font></b> { - filter_lines <font color="#990000">|</font> - process_lines <font color="#990000">|</font> - postprocess_lines <font color="#990000">|</font> - generate_report -} - -main -</pre> -<br /> -<span>The stdout is always passed as a pipe to the next following stage. The stderr is used for info logging.</span><br /> -<br /> -<h3 style='display: inline'>Assign-then-shift</h3><br /> -<br /> -<span>I often refactor existing Bash code. That leads me to add and removing function arguments quite often. It's pretty repetitive work changing the $1, $2.... function argument numbers every time you change the order or add/remove possible arguments.</span><br /> -<br /> -<span>The solution is to use of the "assign-then-shift"-method, which goes like this: "local -r var1=$1; shift; local -r var2=$1; shift". The idea is that you only use "$1" to assign function arguments to named (better readable) local function variables. You will never have to bother about "$2" or above. That is very useful when you constantly refactor your code and remove or add function arguments. It's something that I picked up from a colleague (a pure Bash wizard) some time ago:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#000000">some_function ()</font></b> { - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_foo</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_baz</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_bay</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - - <i><font color="#9A1900"># ...</font></i> -} -</pre> -<br /> -<span>Want to add a param_baz? Just do this:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#000000">some_function ()</font></b> { - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_foo</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_bar</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_baz</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_bay</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - - <i><font color="#9A1900"># ...</font></i> -} -</pre> -<br /> -<span>Want to remove param_foo? Nothing easier than that:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#000000">some_function ()</font></b> { - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_bar</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_baz</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - <b><font color="#0000FF">local</font></b> -r <font color="#009900">param_bay</font><font color="#990000">=</font><font color="#FF0000">"$1"</font><font color="#990000">;</font> <b><font color="#0000FF">shift</font></b> - - <i><font color="#9A1900"># ...</font></i> -} -</pre> -<br /> -<span>As you can see, I didn't need to change any other assignments within the function. Of course, you would also need to change the function argument lists at every occasion where the function is invoked - you would do that within the same refactoring session.</span><br /> -<br /> -<h3 style='display: inline'>Paranoid mode</h3><br /> -<br /> -<span>I call this the paranoid mode. The Bash will stop executing when a command exits with a status not equal to 0:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#0000FF">set</font></b> -e -grep -q foo <font color="#990000"><<<</font> bar -echo Jo -</pre> -<br /> -<span>Here 'Jo' will never be printed out as the grep didn't find any match. It's unrealistic for most scripts to run in paranoid mode purely, so there must be a way to add exceptions. Critical Bash scripts of mine tend to look like this:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><i><font color="#9A1900">#!/usr/bin/env bash</font></i> - -<b><font color="#0000FF">set</font></b> -e - -<b><font color="#000000">some_function ()</font></b> { - <i><font color="#9A1900"># .. some critical code</font></i> - <i><font color="#9A1900"># ...</font></i> - - <b><font color="#0000FF">set</font></b> <font color="#990000">+</font>e - <i><font color="#9A1900"># Grep might fail, but that's OK now</font></i> - grep <font color="#990000">....</font> - <b><font color="#0000FF">local</font></b> -i <font color="#009900">ec</font><font color="#990000">=</font><font color="#009900">$?</font> - <b><font color="#0000FF">set</font></b> -e - - <i><font color="#9A1900"># .. critical code continues ...</font></i> - <b><font color="#0000FF">if</font></b> <font color="#990000">[[</font> <font color="#009900">$ec</font> -ne <font color="#993399">0</font> <font color="#990000">]];</font> <b><font color="#0000FF">then</font></b> - <font color="#990000">:</font> <i><font color="#9A1900"># ...</font></i> - <b><font color="#0000FF">fi</font></b> - <i><font color="#9A1900"># ...</font></i> -} -</pre> -<br /> -<h2 style='display: inline'>Learned</h2><br /> -<br /> -<span>There are also a couple of things I've learned from Google's guide.</span><br /> -<br /> -<h3 style='display: inline'>Unintended lexicographical comparison.</h3><br /> -<br /> -<span>The following looks like a valid Bash code:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#0000FF">if</font></b> <font color="#990000">[[</font> <font color="#FF0000">"${my_var}"</font> <font color="#990000">></font> <font color="#993399">3</font> <font color="#990000">]];</font> <b><font color="#0000FF">then</font></b> - <i><font color="#9A1900"># True for 4, false for 22.</font></i> - do_something -<b><font color="#0000FF">fi</font></b> -</pre> -<br /> -<span>... but it is probably an unintended lexicographical comparison. A correct way would be:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#0000FF">if</font></b> <font color="#990000">((</font> my_var <font color="#990000">></font> <font color="#993399">3</font> <font color="#990000">));</font> <b><font color="#0000FF">then</font></b> - do_something -<b><font color="#0000FF">fi</font></b> -</pre> -<br /> -<span>or</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre><b><font color="#0000FF">if</font></b> <font color="#990000">[[</font> <font color="#FF0000">"${my_var}"</font> -gt <font color="#993399">3</font> <font color="#990000">]];</font> <b><font color="#0000FF">then</font></b> - do_something -<b><font color="#0000FF">fi</font></b> -</pre> -<br /> -<h3 style='display: inline'>PIPESTATUS</h3><br /> -<br /> -<span>I have never used the PIPESTATUS variable before. I knew that it's there, but I never bothered to understand how it works until now thoroughly.</span><br /> -<br /> -<span>The PIPESTATUS variable in Bash allows checking of the return code from all parts of a pipe. If it's only necessary to check the success or failure of the whole pipe, then the following is acceptable:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre>tar -cf - <font color="#990000">./*</font> <font color="#990000">|</font> <font color="#990000">(</font> cd <font color="#FF0000">"${dir}"</font> <font color="#990000">&&</font> tar -xf - <font color="#990000">)</font> -<b><font color="#0000FF">if</font></b> <font color="#990000">((</font> PIPESTATUS<font color="#990000">[</font><font color="#993399">0</font><font color="#990000">]</font> <font color="#990000">!=</font> <font color="#993399">0</font> <font color="#990000">||</font> PIPESTATUS<font color="#990000">[</font><font color="#993399">1</font><font color="#990000">]</font> <font color="#990000">!=</font> <font color="#993399">0</font> <font color="#990000">));</font> <b><font color="#0000FF">then</font></b> - echo <font color="#FF0000">"Unable to tar files to ${dir}"</font> <font color="#990000">>&</font><font color="#993399">2</font> -<b><font color="#0000FF">fi</font></b> -</pre> -<br /> -<span>However, as PIPESTATUS will be overwritten as soon as you do any other command, if you need to act differently on errors based on where it happened in the pipe, you'll need to assign PIPESTATUS to another variable immediately after running the command (don't forget that [ is a command and will wipe out PIPESTATUS).</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre>tar -cf - <font color="#990000">./*</font> <font color="#990000">|</font> <font color="#990000">(</font> cd <font color="#FF0000">"${DIR}"</font> <font color="#990000">&&</font> tar -xf - <font color="#990000">)</font> -<font color="#009900">return_codes</font><font color="#990000">=(</font> <font color="#FF0000">"${PIPESTATUS[@]}"</font> <font color="#990000">)</font> -<b><font color="#0000FF">if</font></b> <font color="#990000">((</font> return_codes<font color="#990000">[</font><font color="#993399">0</font><font color="#990000">]</font> <font color="#990000">!=</font> <font color="#993399">0</font> <font color="#990000">));</font> <b><font color="#0000FF">then</font></b> - do_something -<b><font color="#0000FF">fi</font></b> -<b><font color="#0000FF">if</font></b> <font color="#990000">((</font> return_codes<font color="#990000">[</font><font color="#993399">1</font><font color="#990000">]</font> <font color="#990000">!=</font> <font color="#993399">0</font> <font color="#990000">));</font> <b><font color="#0000FF">then</font></b> - do_something_else -<b><font color="#0000FF">fi</font></b> -</pre> -<br /> -<h2 style='display: inline'>Use common sense and BE CONSISTENT.</h2><br /> -<br /> -<span>The following two paragraphs are thoroughly quoted from the Google guidelines. But they hit the hammer on the head:</span><br /> -<br /> -<span class='quote'>If you are editing code, take a few minutes to look at the code around you and determine its style. If they use spaces around their if clauses, you should, too. If their comments have little boxes of stars around them, make your comments have little boxes of stars around them too.</span><br /> -<br /> -<span class='quote'>The point of having style guidelines is to have a common vocabulary of coding so people can concentrate on what you are saying rather than on how you are saying it. We present global style rules here, so people know the vocabulary. But local style is also important. If the code you add to a file looks drastically different from the existing code around it, the discontinuity throws readers out of their rhythm when they go to read it. Try to avoid this.</span><br /> -<br /> -<br /> -<h2 style='display: inline'>Advanced Bash learning pro tip</h2><br /> -<br /> -<span>I also highly recommend having a read through the "Advanced Bash-Scripting Guide" (not from Google). I use it as the universal Bash reference and learn something new every time I look at it.</span><br /> -<br /> -<a class='textlink' href='https://tldp.org/LDP/abs/html/'>Advanced Bash-Scripting Guide</a><br /> -<br /> -<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> -<br /> -<span>Other related posts are:</span><br /> -<br /> -<a class='textlink' href='./2021-05-16-personal-bash-coding-style-guide.html'>2021-05-16 Personal Bash coding style guide (You are currently reading this)</a><br /> -<a class='textlink' href='./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.html'>2021-06-05 Gemtexter - One Bash script to rule it all</a><br /> -<a class='textlink' href='./2021-11-29-bash-golf-part-1.html'>2021-11-29 Bash Golf Part 1</a><br /> -<a class='textlink' href='./2022-01-01-bash-golf-part-2.html'>2022-01-01 Bash Golf Part 2</a><br /> -<a class='textlink' href='./2023-12-10-bash-golf-part-3.html'>2023-12-10 Bash Golf Part 3</a><br /> -<br /> -<a class='textlink' href='../'>Back to the main site</a><br /> - </div> - </content> - </entry> </feed> diff --git a/gemfeed/index.html b/gemfeed/index.html index b71c6a63..a1635fc7 100644 --- a/gemfeed/index.html +++ b/gemfeed/index.html @@ -12,6 +12,7 @@ <br /> <h2 style='display: inline'>To be in the .zone!</h2><br /> <br /> +<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 - KISS high-availability with OpenBSD</a><br /> <a class='textlink' href='./2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.html'>2024-03-03 - A fine Fyne Android app for quickly logging ideas programmed in Go</a><br /> <a class='textlink' href='./2024-02-04-from-babylon5.buetow.org-to-.cloud.html'>2024-02-04 - From <span class='inlinecode'>babylon5.buetow.org</span> to <span class='inlinecode'>*.buetow.cloud</span></a><br /> <a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 - One reason why I love OpenBSD</a><br /> @@ -10,7 +10,7 @@ <body> <h1 style='display: inline'>foo.zone</h1><br /> <br /> -<span class='quote'>This site was generated at 2024-03-23T16:08:36+02:00 by <span class='inlinecode'>Gemtexter</span></span><br /> +<span class='quote'>This site was generated at 2024-03-30T22:14:16+02:00 by <span class='inlinecode'>Gemtexter</span></span><br /> <br /> <pre> |\---/| @@ -42,6 +42,7 @@ <br /> <h3 style='display: inline'>Posts</h3><br /> <br /> +<a class='textlink' href='./gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 - KISS high-availability with OpenBSD</a><br /> <a class='textlink' href='./gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.html'>2024-03-03 - A fine Fyne Android app for quickly logging ideas programmed in Go</a><br /> <a class='textlink' href='./gemfeed/2024-02-04-from-babylon5.buetow.org-to-.cloud.html'>2024-02-04 - From <span class='inlinecode'>babylon5.buetow.org</span> to <span class='inlinecode'>*.buetow.cloud</span></a><br /> <a class='textlink' href='./gemfeed/2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 - One reason why I love OpenBSD</a><br /> diff --git a/projects.html b/projects.html index 3f0ce184..e82fbfa6 100644 --- a/projects.html +++ b/projects.html @@ -15,19 +15,29 @@ <span>This is a "hidden" site listing my personal project list priorities!</span><br /> <br /> <ul> -<li>Finish and release <span class='inlinecode'>guprecords 2.0.0</span> (Raku)</li> -<li>Blog about Raku and <span class='inlinecode'>guprecords</span> (Raku)</li> -<li>Start blog series about Algorithms in Go (Golang)</li> -<li>Update dependencies and release <span class='inlinecode'>DTail 4.1.1</span> (Golang)</li> -<li>Read "The Go Notebook" and blog about it (Golang)</li> +<li>Read Efficient Go and tune DTail.</li> +<li>Astell & Kern SR35</li> +<li>Kinesis Keyboard</li> +<li>Kiss DNS failover via <span class='inlinecode'>dns-failover.ksh</span> and blog about it (ksh)</li> +<li>Public release of DTail 4.2.3</li> +<li>Work on I/O Riot NG and towards 1.0 (Go, C, Raku)</li> <li>Work through "Bash it out" challenges (Bash)</li> <li>Read "Programming Ruby (3.2)" (Ruby)</li> -<li>Try out <span class='inlinecode'>kickstart.nvim</span> or AstroVim (NeoVim)</li> -<li>Try out <span class='inlinecode'>Ruby LSP</span> in NeoVim, which is not Solar (NeoVim)</li> -<li>Finish <span class='inlinecode'>failunderd 1.0</span> (Perl or Golang)</li> <li>Rewrite site stats script (Perl or Raku)</li> +</ul><br /> +<h2 style='display: inline'>Done</h2><br /> +<br /> +<ul> +<li>Try out <span class='inlinecode'>kickstart.nvim</span> or AstroVim (NeoVim)</li> +<li>Read "The Go Notebook" and blog about it (Golang) [Didnt blog about it]</li> +<li>Try out <span class='inlinecode'>Ruby LSP</span> in NeoVim, which is not Solar (NeoVim) [Switched to Helix editor]</li> +<li>Finish <span class='inlinecode'>failunderd 1.0</span> (Perl or Golang) [done <span class='inlinecode'>dns-failover.ksh</span> now]</li> <li>Modernise my NextCloud and Wallabag infra (Linux, Docker)</li> <li>Release <span class='inlinecode'>Gemtexter 2.1.0</span> and blog about it (Bash)</li> +<li>Update dependencies and release <span class='inlinecode'>DTail 4.1.1</span> (Golang)</li> +<li>Finish and release <span class='inlinecode'>guprecords 2.0.0</span> (Raku)</li> +<li>Blog about Raku and <span class='inlinecode'>guprecords</span> (Raku)</li> +<li>Start blog series about Algorithms in Go (Golang)</li> <p class="footer"> Generated by <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 2.1.0-release</a> | served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | diff --git a/uptime-stats.html b/uptime-stats.html index dc9a3878..5e39b742 100644 --- a/uptime-stats.html +++ b/uptime-stats.html @@ -10,7 +10,7 @@ <body> <h1 style='display: inline'>My machine uptime stats</h1><br /> <br /> -<span class='quote'>This site was last updated at 2024-03-23T16:08:36+02:00</span><br /> +<span class='quote'>This site was last updated at 2024-03-30T22:14:16+02:00</span><br /> <br /> <span>The following stats were collected via <span class='inlinecode'>uptimed</span> on all of my personal computers over many years and the output was generated by <span class='inlinecode'>guprecords</span>, the global uptime records stats analyser of mine.</span><br /> <br /> |