foo.zone feed

KISS high-availability with OpenBSD

2024-03-30T22:12:56+02:00

KISS high-availability with OpenBSD

+
+Published at 2024-03-30T22:12:56+02:00
+
+

+Art by Michael J. Penick (mod. by Paul B.)
+                                               ACME-sky
+        __________
+       / nsd tower\                                             (
+      /____________\                                           (\) awk-ward
+       |:_:_:_:_:_|                                             ))   plant
+       |_:_,--.:_:|                       dig-bubble         (\//   )
+       |:_:|__|_:_|  relayd-castle          _               ) ))   ((
+    _  |_   _  :_:|   _   _   _            (_)             ((((   /)\`
+   | |_| |_| |   _|  | |_| |_| |             o              \\)) (( (
+    \_:_:_:_:/|_|_|_|\:_:_:_:_/             .                ((   ))))
+     |_,-._:_:_:_:_:_:_:_.-,_|                                )) ((//
+     |:|_|:_:_:,---,:_:_:|_|:|                               ,-.  )/
+     |_:_:_:_,'puffy `,_:_:_:_|           _  o               ,;'))((
+     |:_:_:_/  _ | _  \_:_:_:|          (_O                   ((  ))
+_____|_:_:_|  (o)-(o)  |_:_:_|--'`-.     ,--. ksh under-water (((\'/
+ ', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\  /,---.~  goat     \`))
+.  ` |_:_:_|   \`-'/   |_:_:_|.  ` .  `  /()\.__( ) .,-----'`-\(( sed-root
+ ', ;|:_:_:|    `-'    |:_:_:| ', ; ', ; `--'|   \ ', ; ', ; ',')).,--
+.  ` MJP ` .  ` .  ` .  ` . httpd-soil ` .    .  ` .  ` .  ` .  ` .  `
+ ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ;
+
+

+
+I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like (in unsorted and slightly unrelated order) BGP, LVS/IPVS, ldirectord, Pacemaker, STONITH, scripted VIP failover via ARP, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work.
+
+But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space - something as KISS (keep it simple and stupid) as possible.
+
+It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway.
+
+PS: ASCII-art reflects an OpenBSD under-water world with all the tools available in the base system.
+
+

My auto-failover requirements

+
+

Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible.
Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month already!).
It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.
It should be geo-redundant.
It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.
Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.
Let's Encrypt TLS certificates should always work (before and after a failover).
Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover. (This isn't part of the OpenBSD base system, but I coded my own monitoring system in Go.)
Don't configure everything manually. The configuration should be automated and reproducible. (This isn't part of the OpenBSD base system, but I didn't need to install any external software on OpenBSD either.)

My HA solution

+
+

Only OpenBSD base installation required

+
+My HA solution for Web and Gemini is based on DNS (OpenBSD's nsd) and a simple shell script (OpenBSD's ksh and some little sed and awk and grep). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system.
+
+https://man.OpenBSD.org/nsd.8
+https://man.OpenBSD.org/ksh
+https://man.OpenBSD.org/awk
+https://man.OpenBSD.org/sed
+https://man.OpenBSD.org/dig
+https://man.OpenBSD.org/ftp
+https://man.openbsd.org/cron
+
+I also used the dig (for DNS checks) and ftp (for HTTP/HTTPS checks) programs.
+
+The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The ksh script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM:
+
+ +

#!/bin/ksh
+
+ZONES_DIR=/var/nsd/zones/master/
+DEFAULT_MASTER=fishfinger.buetow.org
+DEFAULT_STANDBY=blowfish.buetow.org
+
+determine_master_and_standby () {
+    local master=$DEFAULT_MASTER
+    local standby=$DEFAULT_STANDBY
+
+    .
+    .
+    .
+    
+    local -i health_ok=1
+    if ! ftp -4 -o - https://$master/index.txt | grep -q "Welcome to $master"; then
+        echo "https://$master/index.txt IPv4 health check failed"
+        health_ok=0
+    elif ! ftp -6 -o - https://$master/index.txt | grep -q "Welcome to $master"; then
+        echo "https://$master/index.txt IPv6 health check failed"
+        health_ok=0
+    fi
+    if [ $health_ok -eq 0 ]; then
+        local tmp=$master
+        master=$standby
+        standby=$tmp
+    fi
+
+    .
+    .
+    .
+}
+

+
+The failover scripts looks for the ; Enable failover string in the DNS zone files and swaps the A and AAAA records of the DNS entries accordingly:
+
+ +

fishfinger$ grep failover /var/nsd/zones/master/foo.zone.zone
+        300 IN A 46.23.94.99 ; Enable failover
+        300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover
+www     300 IN A 46.23.94.99 ; Enable failover
+www     300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover
+standby  300 IN A 23.88.35.144 ; Enable failover
+standby  300 IN AAAA 2a01:4f8:c17:20f1::42 ; Enable failover
+

+
+ +

tramsform () {
+	sed -E '
+	/IN A .*; Enable failover/ {
+	    /^standby/! {
+	        s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/master_a)' ; \3/;
+	    }
+	    /^standby/ {
+	        s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/standby_a)' ; \3/;
+	    }
+	}
+	/IN AAAA .*; Enable failover/ {
+	    /^standby/! {
+	        s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/master_aaaa)' ; \3/;
+	    }
+	    /^standby/ {
+	        s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/standby_aaaa)' ; \3/;
+	    }
+	}
+	/ ; serial/ {
+	    s/^( +) ([0-9]+) .*; (.*)/\1 '$(date +%s)' ; \3/;
+	}
+	'
+}
+

+
+After the failover, the script reloads nsd and performs a sanity check to see if DNS still works. If not, a rollback will be performed:
+
+ +

# Race condition (e.g. script execution aborted in the middle of the previous run)
+if [ -f $zone_file.bak ]; then
+    mv $zone_file.bak $zone_file
+fi
+
+cat $zone_file | transform > $zone_file.new.tmp 
+
+grep -v ' ; serial' $zone_file.new.tmp > $zone_file.new.noserial.tmp
+grep -v ' ; serial' $zone_file > $zone_file.old.noserial.tmp
+
+echo "Has zone $zone_file changed?"
+if diff -u $zone_file.old.noserial.tmp $zone_file.new.noserial.tmp; then
+    echo "The zone $zone_file hasn't changed"
+    rm $zone_file.*.tmp
+    return 0
+fi
+
+cp $zone_file $zone_file.bak
+mv $zone_file.new.tmp $zone_file
+rm $zone_file.*.tmp
+echo "Reloading nsd"
+nsd-control reload
+
+if ! zone_is_ok $zone; then
+    echo "Rolling back $zone_file changes"
+    cp $zone_file $zone_file.invalid
+    mv $zone_file.bak $zone_file
+    echo "Reloading nsd"
+    nsd-control reload
+    zone_is_ok $zone
+    return 3
+fi
+
+for cleanup in invalid bak; do
+    if [ -f $zone_file.$cleanup ]; then
+        rm $zone_file.$cleanup
+    fi
+done
+
+echo "Failover of zone $zone to $MASTER completed"
+return 1
+

+
+A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output.
+
+The authorative nameserver for my domains runs on both VMs, and both are configured to be a "master" DNS server so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case.
+
+Check out the whole script here:
+
+dns-failover.ksh
+
+

Fairly cheap and geo-redundant

+
+I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany).
+
+https://openbsd.amsterdam
+https://www.hetzner.cloud
+
+I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required.
+
+A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions.
+
+

Failover time and split-brain

+
+A DNS failover doesn't happen immediately. I've configured a DNS TTL of 300 seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover).
+
+A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated.
+
+

Failover support for multiple protocols

+
+With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's httpd (OpenBSD's HTTP server) and relayd (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entry, which hosts receive the requests.
+
+https://man.openbsd.org/httpd.8
+https://man.openbsd.org/relayd.8
+
+For example, the master is responsible for the https://www.foo.zone and https://foo.zone hosts, whereas the standby can be reached via https://standby.foo.zone (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. irregular.ninja, paul.buetow.org and so on. The same applies to my Gemini capsules for gemini://foo.zone, gemini://standby.foo.zone, gemini://paul.buetow.org and gemini://standby.paul.buetow.org.
+
+On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)!
+
+

Let's encrypt TLS certificates

+
+All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate.
+
+If the master always serves foo.zone and the standby always standby.foo.zone, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for foo.zone and the new standby wouldn't have a valid certificate for standby.foo.zone which would lead to TLS errors on the clients.
+
+As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that:
+
+

In an odd week number, the first server is the default master
In an even week number, the second server is the default master.

+Which translates to:
+
+ +

# Weekly auto-failover for Let's Encrypt automation
+local -i -r week_of_the_year=$(date +%U)
+if [ $(( week_of_the_year % 2 )) -eq 0 ]; then
+    local tmp=$master
+    master=$standby
+    standby=$tmp
+fi
+

+
+This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM.
+
+The ACME automation is yet another daily CRON script /usr/local/bin/acme.sh. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt.
+
+Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty.
+
+acme.sh.tpl - Rex template for the acme.sh script of mine.
+https://man.openbsd.org/acme-client.1
+Let's Encrypt with OpenBSD and Rex
+
+

Monitoring

+
+CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed.
+
+https://codeberg.org/snonux/gogios
+KISS server monitoring with Gogios
+
+

Rex automation

+
+I use Rexify, a friendly configuration management system that allows automatic deployment and configuration.
+
+https://www.rexify.org
+codeberg.org/snonux/rexfiles/frontends
+
+

More HA

+
+Other high-available services running on my OpenBSD VMs are my MTAs for mail forwarding (OpenSMTPD) and the authoritative DNS servers (nsd) for all my domains. No particular HA setup is required, though, as the protocols (SMTP and DNS) already take care of the failover to the next available host!
+
+As a password manager, I use geheim, a command-line tool I wrote in Ruby with encrypted files in a git repository (I even have it installed in Termux on my Phone). For HA reasons, I simply updated the client code so that it always synchronises the database with both servers when I run the sync command there.
+
+https://codeberg.org/snonux/geheim
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Other *BSD and KISS related posts are:
+
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2022-10-30 Installing DTail on OpenBSD
+2023-06-01 KISS server monitoring with Gogios
+2023-10-29 KISS static web photo albums with photoalbum.sh
+2024-01-13 One reason why I love OpenBSD
+2024-04-01 KISS high-availability with OpenBSD (You are currently reading this)
+
+Back to the main site
+

A fine Fyne Android app for quickly logging ideas programmed in Go

2024-03-03T00:07:21+02:00

A fine Fyne Android app for quickly logging ideas programmed in Go

+
+Published at 2024-03-03T00:07:21+02:00
+
+I am an ideas person. I find myself frequently somewhere on the streets with an idea in my head but no paper journal noting it down.
+
+I have tried many note apps for my Android (I use GrapheneOS) phone. Most of them either don't do what I want, are proprietary software, require Google Play services (I have the main profile on my phone de-googled) or are too bloated. I was never into mobile app development, as I'm not too fond of the complexity of the developer toolchains. I don't want to use Android Studio (as a NeoVim user), and I don't want to use Java or Kotlin. I want to use a language I know (and like) for mobile app development. Go would be one of those languages.
+
+
+
+Enter Quick logger – a compact GUI Android (well, cross-platform due to Fyne) app I've crafted using Go and the nifty Fyne framework. With Fyne, the app can be compiled easily into an Android APK. As of this writing, this app's whole Go source code is only 75 lines short!! This little tool is designed for spontaneous moments, allowing me to quickly log my thoughts as plain text files on my Android phone. There are no fancy file formats. Just plain text!
+
+https://codeberg.org/snonux/quicklogger
+https://fyne.io
+https://go.dev
+
+There's no need to navigate complex menus or deal with sync issues. I jot down my Idea, and Quick logger saves it to a plain text file in a designated local folder on my phone. There is one text file per note (timestamp in the file name). Once logged, the file can't be edited anymore (it keeps it simple). If I want to correct or change a note, I simply write a new one. My notes are always small (usually one short sentence each), so there isn't the need for an edit functionality. I can edit them later on my actual computer if I want to.
+
+With Syncthing, the note files are then synchronised to my home computer to my ~/Notes directory. From there, a small glue Raku script adds them to my Taskwarrior DB so that I can process them later (e.g. take action on that one Idea I had). That then will delete the original note files from my computer and also (through Syncthing) from my phone.
+
+https://syncthing.net
+https://raku.org
+https://taskwarrior.org
+
+Quick logger's user interface is as minimal as it gets. When I launch Quick logger, I'm greeted with a simple window where I can type plain text. Hit the "Log text" button, and voilà – the input is timestamped and saved as a file in my chosen directory. If I need to change the directory, the "Preferences" button brings up a window where I can set the notes folder and get back to logging.
+
+For the code-savvy folks out there, Quick logger is a neat example of what you can achieve with Go and Fyne. It's a testament to building functional, cross-platform apps without getting bogged down in the nitty-gritty of platform-specific details. Thanks to Fyne, I am pleased with how easy it is to make mobile Android apps in Go.
+
+
+
+My Android apps will never be polished, but they will get the job done, and this is precisely how I want them to be. Minimalistic but functional. I could spend more time polishing Quick logger, but my Quick logger app then may be the same as any other notes app out there (complicated or bloated).
+
+

All easy-peasy?

+
+I did have some issues with the app logo for Android, though. Android always showed the default app icon and not my custom icon whenever I used a custom AndroidManifest.xml for custom app storage permissions. Without a custom AndroidAmnifest.xml the app icon would be displayed under Android, but then the app would not have the MANAGE_EXTERNAL_STORAGE permission, which is required for Quick logger to write to a custom directory. I found a workaround, which I commented on here at Github:
+
+https://github.com/fyne-io/fyne/issues/3077#issuecomment-1912697360
+
+What worked however (app icon showing up) was to clone the fyne project, change the occurances of android.permission.INTERNET to android.permission.MANAGE_EXTERNAL_STORAGE (as these are all the changes I want in my custom android manifest) in the source tree, re-compile fyne. Now all works. I know, this is more of an hammer approach!
+
+Hopefully, I won't need to use this workaround anymore. But for now, it is a fair tradeoff for what I am getting.
+
+I hope this will inspire you to write your own small mobile apps in Go using the awesome Fyne framework! PS: The Quick logger logo was generated by ChatGPT.
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Other Go related posts are:
+
+2023-04-09 Algorithms and Data Structures in Go - Part 1
+2024-03-03 A fine Fyne Android app for quickly logging ideas programmed in Go (You are currently reading this)
+
+Back to the main site
+

From `babylon5.buetow.org` to `*.buetow.cloud`

2024-02-04T00:50:50+02:00

From babylon5.buetow.org to *.buetow.cloud

+
+Published at 2024-02-04T00:50:50+02:00
+
+Recently, my employer sent me to a week-long AWS course. After the course, there wasn't any hands-on project I could dive into immediately, so I moved parts of my personal infrastructure to AWS to level up a bit through practical hands-on.
+
+So, I migrated all of my Docker-based self-hosted services to AWS. Usually, I am not a big fan of big cloud providers and instead use smaller hosters or indie providers and self-made solutions. However, I also must go with the times and try out technologies currently hot on the job market. I don't want to become the old man who yells at cloud :D
+
+
+
+

The old *.buetow.org way

+
+Before the migration, all those services were reachable through buetow.org-subdomains (Buetow is my last name) and ran on Docker containers on a single Rocky Linux 9 VM at Hetzner. And there was a Nginx reverse proxy with TLS offloading (with Let's Encrypt certificates). The Rocky Linux 9's hostname was babylon5.buetow.org (based on the Science Fiction series).
+
+https://en.wikipedia.org/wiki/Babylon_5
+
+The downsides of this setup were:
+
+

Not highly available. If the server goes down, no service is reachable until it's repaired. To be fair, the Hetzner cloud VM is redundant by itself and would have re-spawned on a different worker node, I suppose.
Manual installation.

+About the manual installation part: I could have used a configuration management system like Rexify, Puppet, etc. But I decided against it back in time, as setting up Docker containers isn't so complicated through simple start scripts. And it's only a single Linux box where a manual installation is less painful. However, regular backups (which Hetzner can do automatically for you) were a must.
+
+The benefits of this setup were:
+
+

KISS (Keep it Simple Stupid)
Cheap

I kept my buetow.org OpenBSD boxes alive

+
+As pointed out, I only migrated the Docker-based self-hosted services (which run on the Babylon 5 Rocky Linux box) to AWS. Many self-hostable apps come with ready-to-use container images, making deploying them easy.
+
+My other two OpenBSD VMs (blowfish.buetow.org, hosted at Hetzner, and fishfinger.buetow.org, hosted at OpenBSD Amsterdam) still run (and they will keep running) the following services:
+
+

HTTP server for my websites (e.g. https://foo.zone, ...)
ACME for Let's Encrypt TLS certificate auto-renewal.
Gemini server for my capsules (e.g. gemini://foo.zone)
Authoritative DNS servers for my domains (but buetow.cloud, which is on Route 53 now)
Mail transfer agent (MTA)
My Gogios monitoring system.
My IRC bouncer.

+It is all automated with Rex, aka Rexify. This OpenBSD setup is my "fun" or "for pleasure" setup. Whereas the Rocky Linux 9 one I always considered the "pratical means to the end"-setup to have 3rd party Docker containers up and running with as little work as possible.
+
+(R)?ex, the friendly automation framework
+KISS server monitoring with Gogios
+Let's encrypt with OpenBSD and Rex
+
+

The new *.buetow.cloud way

+
+With AWS, I decided to get myself a new domain name, as I could fully separate my AWS setup from my conventional setup and give Route 53 as an authoritative DNS a spin.
+
+I decided to automate everything with Terraform, as I wanted to learn to use it as it appears standard now in the job market.
+
+All services are installed automatically to AWS ECS Fargate. ECS is AWS's Elastic Container Service, and Fargate automatically manages the underlying hardware infrastructure (e.g., how many CPUs, RAM, etc.) for me. So I don't have to bother about having enough EC2 instances to serve my demands, for example.
+
+The authoritative DNS for the buetow.cloud domain is AWS Route 53. TLS certificates are free here at AWS and offloaded through the AWS Application Load Balancer. The LB acts as a proxy to the ECS container instances of the services. A few services I run in ECS Fargate also require the AWS Network Load Balancer.
+
+All services require some persistent storage. For that, I use an encrypted EFS file system, automatically replicated across all AZs (availability zones) of my region of choice, eu-central-1.
+
+In case of an AZ outage, I could re-deploy all the failed containers in another AZ, and all the data would still be there.
+
+The EFS automatically gets backed up by AWS for me following their standard Backup schedule. The daily backups are kept for 30 days.
+
+Domain registration, TLS certificate configuration and configuration of the EFS backup were quickly done through the AWS web interface. These were only one-off tasks, so they weren't fully automated through Terraform.
+
+You can find all Terraform manifests here:
+
+https://codeberg.org/snonux/terraform
+
+Whereas:
+
+

org-buetow-base sets up the bare VPC (IPv4 and IPv6 subnets in 3 AZs, EFS, ECR (the AWS container registry for some self-built containers) and Route 53 zone. It's the requirement for most other Terraform manifests in this repository.
org-buetow-bastion sets up a minimal Amazon Linux EC2 instance where I can manually SSH into and look at the EFS file system (if required).
org-buetow-elb sets up the Elastic Load Balancer, a prerequisite for any service running in ECS Fargate.
org-buetow-ecs finally sets up and deploys all the Docker apps mentioned above. Any apps can be turned on or off via the variables.tf file.

The container apps

+
+And here, finally, is the list of all the container apps my Terraform manifests deploy. The FQDNs here may not be reachable. I spin them up only on demand (for cost reasons). All services are fully dual-stacked (IPv4 & IPv6).
+
+

flux.buetow.cloud

+
+Miniflux is a minimalist and opinionated feed reader. With the move to AWS, I also retired my bloated instance of NextCloud. So, with Miniflux, I retired from NextCloud News.
+
+Miniflux requires two ECS containers. One is the Miniflux app, and the other is the PostgreSQL DB.
+
+https://miniflux.app/
+
+
+

audiobookshelf.buetow.cloud

+
+Audiobookshelf was the first Docker app I installed. It is a Self-hosted audiobook and podcast server. It comes with a neat web interface, and there is also an Android app available, which works also in offline mode. This is great, as I only have the ECS instance sometimes running for cost savings.
+
+With Audiobookshelf, I replaced my former Audible subscription and my separate Podcast app. For Podcast synchronisation I used to use the Gpodder NextCloud sync app. But that one I retired now with Audiobookshelf as well :-)
+
+https://www.audiobookshelf.org
+
+

syncthing.buetow.cloud

+
+Syncthing is a continuous file synchronisation program. In real-time, it synchronises files between two or more computers, safely protected from prying eyes. Your data is your own, and you deserve to choose where it is stored, whether it is shared with some third party, and how it's transmitted over the internet.
+
+With Syncthing, I retired my old NextCloud Files and file sync client on all my devices. I also quit my NextCloud Notes setup. All my Notes are now plain Markdown files in a Notes directory. On Android, I can edit them with any text or Markdown editor (e.g. Obsidian), and they will be synchronised via Syncthing to my other computers, both forward and back.
+
+I use Syncthing to synchronise some of my Phone's data (e.g. Notes, Pictures and other documents). Initially, I synced all of my pictures, videos, etc., with AWS. But that was pretty expensive. So for now, I use it only whilst travelling. Otherwise, I will use my Syncthing instance here on my LAN (I have a cheap cloud backup in AWS S3 Glacier Deep Archive, but that's for another blog post).
+
+https://syncthing.net/
+
+

radicale.buetow.cloud

+
+Radicale is an excellent minimalist WebDAV calendar and contact synchronisation server. It was good enough to replace my NextCloud Calendar and NextCloud Contacts setup. Unfortunately, there wasn't a ready-to-use Docker image. So, I created my own.
+
+On Android, it works great together with the DAVx5 client for synchronisation.
+
+https://radicale.org/
+https://codeberg.org/snonux/docker-radicale-server
+https://www.davx5.com/
+
+

bag.buetow.cloud

+
+Wallabag is a self-hostable "save now - read later" service, and it also comes with an Android app which also has an offline mode. Think of Getpocket, but open-source!
+
+https://wallabag.org/
+https://github.com/wallabag/wallabag
+
+

anki.buetow.cloud

+
+Anki is a great (the greatest) flash-card learning program. I am currently learning Bulgarian as my 3rd language. There is also an Android app that has an offline mode, and advanced users can also self-host the server anki-sync-server. For some reason (not going into the details here), I had to build my own Docker image for the server.
+
+https://apps.ankiweb.net/
+https://codeberg.org/snonux/docker-anki-sync-server
+
+

vault.buetow.cloud

+
+Vaultwarden is an alternative implementation of the Bitwarden server API written in Rust and compatible with upstream Bitwarden clients, perfect for self-hosted deployment where running the official resource-heavy service might not be ideal. So, this is a great password manager server which can be used with any Bitwarden Android app.
+
+I currently don't use it, but I may in the future. I made it available in my ECS Fargate setup anyway for now.
+
+https://github.com/dani-garcia/vaultwarden
+
+I currently use geheim, a Ruby command line tool I wrote, as my current password manager. You can read a little bit about it here under "More":
+
+Sweating the small stuff
+
+

bastion.buetow.cloud

+
+This is a tiny ARM-based Amazon Linux EC2 instance, which I sometimes spin up for investigation or manual work on my EFS file system in AWS.
+
+

Conclusion

+
+I have learned a lot about AWS and Terraform during this migration. This was actually my first AWS hands-on project with practical use.
+
+All of this was not particularly difficult (but at times a bit confusing). I see the use of Terraform managing more extensive infrastructures (it was even helpful for my small setup here). At least I know now what all the buzz is about :-). I don't think Terraform's HCL is a nice language. It get's it's job done, but it could be more elegant IMHO.
+
+Deploying updates to AWS are much easier, and some of the manual maintenance burdens of my Rocky Linux 9 VM are no longer needed. So I will have more time for other projects!
+
+Will I keep it in the cloud? I don't know yet. But maybe I won't renew the buetow.cloud domain and instead will use *.cloud.buetow.org or *.aws.buetow.org subdomains.
+
+Will the AWS setup be cheaper than my old Rocky Linux setup? It might be more affordable as I only turn ECS and the load balancers on or off on-demand. Time will tell! The first forecasts suggest that it will be around the same costs.
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Back to the main site
+

One reason why I love OpenBSD

2024-01-13T22:55:33+02:00

One reason why I love OpenBSD

+
+Published at 2024-01-13T22:55:33+02:00
+
+

+           FISHKISSFISHKIS               
+       SFISHKISSFISHKISSFISH            F
+    ISHK   ISSFISHKISSFISHKISS         FI
+  SHKISS   FISHKISSFISHKISSFISS       FIS
+HKISSFISHKISSFISHKISSFISHKISSFISH    KISS
+  FISHKISSFISHKISSFISHKISSFISHKISS  FISHK
+      SSFISHKISSFISHKISSFISHKISSFISHKISSF
+  ISHKISSFISHKISSFISHKISSFISHKISSF  ISHKI
+SSFISHKISSFISHKISSFISHKISSFISHKIS    SFIS
+  HKISSFISHKISSFISHKISSFISHKISS       FIS
+    HKISSFISHKISSFISHKISSFISHK         IS
+       SFISHKISSFISHKISSFISH            K
+         ISSFISHKISSFISHK               
+

+
+I just upgraded my OpenBSD's from 7.3 to 7.4 by following the unattended upgrade guide:
+
+https://www.openbsd.org/faq/upgrade74.html
+
+ +

$ doas installboot sd0 # Update the bootloader (not for every upgrade required)
+$ doas sysupgrade # Update all binaries (including Kernel)
+

+
+sysupgrade downloaded and upgraded to the next release and rebooted the system. After the reboot, I run:
+
+ +

$ doas sysmerge # Update system configuration files
+$ doas pkg_add -u # Update all packages
+$ doas reboot # Just in case, reboot one more time
+

+
+That's it! Took me around 5 minutes in total! No issues, only these few comands, only 5 minutes! It just works! No problems, no conflicts, no tons (actually none) config file merge conflicts.
+
+I followed the same procedure the previous times and never encountered any difficulties with any OpenBSD upgrades.
+
+I have seen upgrades of other Operating Systems either take a long time or break the system (which takes manual steps to repair). That's just one of many reasons why I love OpenBSD! There appear never to be any problems. It just gets its job done!
+
+The OpenBSD Project
+
+BTW: are you looking for an opinionated OpenBSD VM hoster? OpenBSD Amsterdam may be for you. They rock (I am having a VM there, too)!
+
+https://openbsd.amsterdam
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Other *BSD related posts are:
+
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2022-10-30 Installing DTail on OpenBSD
+2024-01-13 One reason why I love OpenBSD (You are currently reading this)
+2024-04-01 KISS high-availability with OpenBSD
+
+Back to the main site
+

Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect

2024-01-09T18:35:48+02:00

Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect

+
+Published at 2024-01-09T18:35:48+02:00
+
+This is the third part of my Site Reliability Engineering (SRE) series. I am currently employed as a Site Reliability Engineer and will try to share what SRE is about in this blog series.
+
+2023-08-18 Site Reliability Engineering - Part 1: SRE and Organizational Culture
+2023-11-19 Site Reliability Engineering - Part 2: Operational Balance in SRE
+2024-01-09 Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect (You are currently reading this)
+
+

+                    ..--""""----..                 
+                 .-"   ..--""""--.j-.              
+              .-"   .-"        .--.""--..          
+           .-"   .-"       ..--"-. \/    ;         
+        .-"   .-"_.--..--""  ..--'  "-.  :         
+      .'    .'  /  `. \..--"" __ _     \ ;         
+     :.__.-"    \  /        .' ( )"-.   Y          
+     ;           ;:        ( )     ( ).  \         
+   .':          /::       :            \  \        
+ .'.-"\._   _.-" ; ;      ( )    .-.  ( )  \       
+  "    `."""  .j"  :      :      \  ;    ;  \      
+    bug /"""""/     ;      ( )    "" :.( )   \     
+       /\    /      :       \         \`.:  _ \    
+      :  `. /        ;       `( )     (\/ :" \ \   
+       \   `.        :         "-.(_)_.'   t-'  ;  
+        \    `.       ;                    ..--":  
+         `.    `.     :              ..--""     :  
+           `.    "-.   ;       ..--""           ;  
+             `.     "-.:_..--""            ..--"   
+               `.      :             ..--""        
+                 "-.   :       ..--""              
+                    "-.;_..--""                    
+
+

+
+

On-Call Culture and the Human Aspect: Prioritising Well-being in the Realm of Reliability

+
+Site Reliability Engineering is synonymous with ensuring system reliability, but the human factor is an often-underestimated part of this discipline. Ensuring an healthy on-call culture is as critical as any technical solution. The well-being of the engineers is an important factor.
+
+Firstly, a healthy on-call rotation is about more than just managing and responding to incidents. It's about the entire ecosystem that supports this practice. This involves reducing pain points, offering mentorship, rapid iteration, and ensuring that engineers have the right tools and processes. One ceavat is, that engineers should be willing to learn. Especially in on-call rotation embedding SREs with other engineers (for example Software Engineers or QA Engineers), it's difficult to motivate everyone to engage. QA Engineers want to test the software, Software Engineers want to implement new features; they don't want to troubleshoot and debug production incidents. It can be depressing for the mentoring SRE.
+
+Furthermore, the metrics that measure the success of an on-call experience are only sometimes straightforward. While one might assume that fewer pages translate to better on-call expertise (which is true to a degree, as who wants to receive a page out of office hours?), it's not always the volume of pages that matters most. Trust, ownership, accountability, and effective communication play the important roles.
+
+An important part is giving feedback about the on-call experience to ensure continuous learning. If alerts are mostly noise, they should be tuned or even eliminated. If alerts are actionable, can recurring tasks be automated? If there are knowledge gaps, is the documentation not good enough? Continuous retrospection ensures that not only do systems evolve, but the experience for the on-call engineers becomes progressively better.
+
+Onboarding for on-call duties is a crucial aspect of ensuring the reliability and efficiency of systems. This process involves equipping new team members with the knowledge, tools, and support to handle incidents confidently. It begins with an overview of the system architecture and common challenges, followed by training on monitoring tools, alerting mechanisms, and incident response protocols. Shadowing experienced on-call engineers can offer practical exposure. Too often, new engineers are thrown into the cold water without proper onboarding and training because the more experienced engineers are too busy fire-fighting production issues in the first place.
+
+An always-on, always-alert culture can lead to burnout. Engineers should be encouraged to recognise their limits, take breaks, and seek support when needed. This isn't just about individual health; a burnt-out engineer can have cascading effects on the entire team and the systems they manage. A successful on-call culture ensures that while systems are kept running, the engineers are kept happy, healthy, and supported. The more experienced engineers should take time to mentor the junior engineers, but the junior engineers should also be fully engaged, try to investigate and learn new things by themselves.
+
+For the junior engineer, it's too easy to fall back and ask the experts in the team every time an issue arises. This seems reasonable, but serving recipes for solving production issues on a silver tablet won't scale forever, as there are infinite scenarios of how production systems can break. So every engineer should learn to debug, troubleshoot and resolve production incidents independently. The experts will still be there for guidance and step in when the junior gets stuck after trying, but the experts should also learn to step down so that lesser experienced engineers can step up and learn. But mistakes can always happen here; that's why having a blameless on-call culture is essential.
+
+A blameless on-call culture is a must for a safe and collaborative environment where engineers can effectively respond to incidents without fear of retribution. This approach acknowledges that mistakes are a natural part of the learning and innovation process. When individuals are assured they won't be punished for errors, they're more likely to openly discuss mistakes, allowing the entire team to learn and grow from each incident. Furthermore, a blameless culture promotes psychological safety, enhances job satisfaction, reduces burnout, and ensures that talent remains committed and engaged.
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Back to the main site
+

Bash Golf Part 3

2023-12-10T11:35:54+02:00

Bash Golf Part 3

+
+Published at 2023-12-10T11:35:54+02:00
+
+

+    '\       '\        '\                   .  .          |>18>>
+      \        \         \              .         ' .     |
+     O>>      O>>       O>>         .                 'o  |
+      \       .\. ..    .\. ..   .                        |
+      /\    .  /\     .  /\    . .                        |
+     / /   .  / /  .'.  / /  .'    .                      |
+jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+                        Art by Joan Stark, mod. by Paul Buetow
+

+
+This is the third blog post about my Bash Golf series. This series is random Bash tips, tricks, and weirdnesses I have encountered over time.
+
+2021-11-29 Bash Golf Part 1
+2022-01-01 Bash Golf Part 2
+2023-12-10 Bash Golf Part 3 (You are currently reading this)
+
+

FUNCNAME

+
+FUNCNAME is an array you are looking for a way to dynamically determine the name of the current function (which could be considered the callee in the context of its own execution), you can use the special variable FUNCNAME. This is an array variable that contains the names of all shell functions currently in the execution call stack. The element FUNCNAME[0] holds the name of the currently executing function, FUNCNAME[1] the name of the function that called that, and so on.
+
+This is particularly useful for logging when you want to include the callee function in the log output. E.g. look at this log helper:
+
+ +

#!/usr/bin/env bash
+
+log () {
+    local -r level="$1"; shift
+    local -r message="$1"; shift
+    local -i pid="$$"
+
+    local -r callee=${FUNCNAME[1]}
+    local -r stamp=$(date +%Y%m%d-%H%M%S)
+
+    echo "$level|$stamp|$pid|$callee|$message" >&2
+}
+
+at_home_friday_evening () {
+    log INFO 'One Peperoni Pizza, please'
+}
+
+at_home_friday_evening
+

+
+The output is as follows:
+
+ +

❯ ./logexample.sh
+INFO|20231210-082732|123002|at_home_friday_evening|One Peperoni Pizza, please
+

+
+

:(){ :|:& };:

+
+This one may be widely known already, but I am including it here as I found a cute image illustrating it. But to break :(){ :|:& };: down:
+
+

:(){ } is really a declaration of the function :
The ; is ending the current statement
The : at the end is calling the function :
:|:& is the function body

+Let's break down the function body :|:&:
+
+

The first : is calling the function recursively
The |: is piping the output to the function : again (parallel recursion)
The & lets it run in the background.

+So, it's a fork bomb. If you run it, your computer will run out of resources eventually. (Modern Linux distributions could have reasonable limits configured for your login session, so it won't bring down your whole system anymore unless you run it as root!)
+
+And here is the cute illustration:
+
+
+
+

Inner functions

+
+Bash defines variables as it is interpreting the code. The same applies to function declarations. Let's consider this code:
+
+ +

#!/usr/bin/env bash
+
+outer() {
+  inner() {
+    echo 'Intel inside!'
+  }
+  inner
+}
+
+inner
+outer
+inner
+

+
+And let's execute it:
+
+

+❯ ./inner.sh
+/tmp/inner.sh: line 10: inner: command not found
+Intel inside!
+Intel inside!
+

+
+What happened? The first time inner was called, it wasn't defined yet. That only happens after the outer run. Note that inner will still be globally defined. But functions can be declared multiple times (the last version wins):
+
+ +

#!/usr/bin/env bash
+
+outer1() {
+  inner() {
+    echo 'Intel inside!'
+  }
+  inner
+}
+
+outer2() {
+  inner() {
+    echo 'Wintel inside!'
+  }
+  inner
+}
+
+outer1
+inner
+outer2
+inner
+

+
+And let's run it:
+
+

+❯ ./inner2.sh
+Intel inside!
+Intel inside!
+Wintel inside!
+Wintel inside!
+

+
+

Exporting functions

+
+Have you ever wondered how to execute a shell function in parallel through xargs? The problem is that this won't work:
+
+ +

#!/usr/bin/env bash
+
+some_expensive_operations() {
+  echo "Doing expensive operations with '$1' from pid $$"
+}
+
+for i in {0..9}; do echo $i; done \
+  | xargs -P10 -I{} bash -c 'some_expensive_operations "{}"'
+

+
+We try here to run ten parallel processes; each of them should run the some_expensive_operations function with a different argument. The arguments are provided to xargs through STDIN one per line. When executed, we get this:
+
+

+❯ ./xargs.sh
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+bash: line 1: some_expensive_operations: command not found
+

+
+There's an easy solution for this. Just export the function! It will then be magically available in any sub-shell!
+
+ +

#!/usr/bin/env bash
+
+some_expensive_operations() {
+  echo "Doing expensive operations with '$1' from pid $$"
+}
+export -f some_expensive_operations
+
+for i in {0..9}; do echo $i; done \
+  | xargs -P10 -I{} bash -c 'some_expensive_operations "{}"'
+

+
+When we run this now, we get:
+
+

+❯ ./xargs.sh
+Doing expensive operations with '0' from pid 132831
+Doing expensive operations with '1' from pid 132832
+Doing expensive operations with '2' from pid 132833
+Doing expensive operations with '3' from pid 132834
+Doing expensive operations with '4' from pid 132835
+Doing expensive operations with '5' from pid 132836
+Doing expensive operations with '6' from pid 132837
+Doing expensive operations with '7' from pid 132838
+Doing expensive operations with '8' from pid 132839
+Doing expensive operations with '9' from pid 132840
+

+
+If some_expensive_function would call another function, the other function must also be exported. Otherwise, there will be a runtime error again. E.g., this won't work:
+
+ +

#!/usr/bin/env bash
+
+some_other_function() {
+  echo "$1"
+}
+
+some_expensive_operations() {
+  some_other_function "Doing expensive operations with '$1' from pid $$"
+}
+export -f some_expensive_operations
+
+for i in {0..9}; do echo $i; done \
+  | xargs -P10 -I{} bash -c 'some_expensive_operations "{}"'
+

+
+... because some_other_function isn't exported! You will also need to add an export -f some_other_function!
+
+

Dynamic variables with local

+
+You may know that local is how to declare local variables in a function. Most don't know that those variables actually have dynamic scope. Let's consider the following example:
+
+ +

#!/usr/bin/env bash
+
+foo() {
+  local foo=bar # Declare local/dynamic variable
+  bar
+  echo "$foo"
+}
+
+bar() {
+  echo "$foo"
+  foo=baz
+}
+
+foo=foo # Declare global variable
+foo # Call function foo
+echo "$foo"
+

+
+Let's pause a minute. What do you think the output would be?
+
+Let's run it:
+
+

+❯ ./dynamic.sh
+bar
+baz
+foo
+

+
+What happened? The variable foo (declared with local) is available in the function it was declared in and in all other functions down the call stack! We can even modify the value of foo, and the change will be visible up the call stack. It's not a global variable; on the last line, echo "$foo" echoes the global variable content.
+
+
+

if conditionals

+
+Consider all variants here more or less equivalent:
+
+ +

#!/usr/bin/env bash
+
+declare -r foo=foo
+declare -r bar=bar
+
+if [ "$foo" = foo ]; then
+  if [ "$bar" = bar ]; then
+    echo ok1
+  fi
+fi
+
+if [ "$foo" = foo ] && [ "$bar" == bar ]; then
+  echo ok2a
+fi
+
+[ "$foo" = foo ] && [ "$bar" == bar ] && echo ok2b
+
+if [[ "$foo" = foo && "$bar" == bar ]]; then
+  echo ok3a
+fi
+
+ [[ "$foo" = foo && "$bar" == bar ]] && echo ok3b
+
+if test "$foo" = foo && test "$bar" = bar; then
+  echo ok4a
+fi
+
+test "$foo" = foo && test "$bar" = bar && echo ok4b
+

+
+The output we get is:
+
+

+❯ ./if.sh
+ok1
+ok2a
+ok2b
+ok3a
+ok3b
+ok4a
+ok4b
+

+
+

Multi-line comments

+
+You all know how to comment. Put a # in front of it. You could use multiple single-line comments or abuse heredocs and redirect it to the : no-op command to emulate multi-line comments.
+
+ +

#!/usr/bin/env bash
+
+# Single line comment
+
+# These are two single line
+# comments one after another
+
+: <<COMMENT
+This is another way a
+multi line comment
+could be written!
+COMMENT
+

+
+I will not demonstrate the execution of this script, as it won't print anything! It's obviously not the most pretty way of commenting on your code, but it could sometimes be handy!
+
+

Don't change it while it's executed

+
+Consider this script:
+
+ +

#!/usr/bin/env bash
+
+echo foo
+echo echo baz >> $0
+echo bar
+

+
+When it is run, it will do:
+
+

+❯ ./if.sh
+foo
+bar
+baz
+❯ cat if.sh
+#!/usr/bin/env bash
+
+echo foo
+echo echo baz >> $0
+echo bar
+echo baz
+

+
+So what happened? The echo baz line was appended to the script while it was still executed! And the interpreter also picked it up! It tells us that Bash evaluates each line as it encounters it. This can lead to nasty side effects when editing the script while it is still being executed! You should always keep this in mind!
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Other related posts are:
+
+2021-05-16 Personal Bash coding style guide
+2021-06-05 Gemtexter - One Bash script to rule it all
+2021-11-29 Bash Golf Part 1
+2022-01-01 Bash Golf Part 2
+2023-12-10 Bash Golf Part 3 (You are currently reading this)
+
+Back to the main site
+