foo.zone feed

Terminal multiplexing with `tmux` - Z-Shell edition

2024-06-23T22:41:59+03:00

Terminal multiplexing with tmux - Z-Shell edition

-
-Published at 2024-06-23T22:41:59+03:00; Last updated 2025-05-02
-
-This is the Z-Shell version. There is also a Fish version:
-
-./2025-05-02-terminal-multiplexing-with-tmux-fish-edition.html
-
-Tmux (Terminal Multiplexer) is a powerful, terminal-based tool that manages multiple terminal sessions within a single window. Here are some of its primary features and functionalities:
-
-

Session management
Window and Pane management
Persistent Workspace
Customization

-https://github.com/tmux/tmux/wiki
-
-

-         _______
-        |.-----.|
-        || Tmux||
-        ||_.-._||
-        `--)-(--`
-       __[=== o]___
-      |:::::::::::|\
-jgs   `-=========-`()
-    mod. by Paul B.
-

-
-

Terminal multiplexing with tmux - Z-Shell edition
⇢ Before continuing...
⇢ Shell aliases
⇢ The tn alias - Creating a new session
⇢ ⇢ Cleaning up default sessions automatically
⇢ ⇢ Renaming sessions
⇢ The ta alias - Attaching to a session
⇢ The tr alias - For a nested remote session
⇢ ⇢ Change of the Tmux prefix for better nesting
⇢ The ts alias - Searching sessions with fuzzy finder
⇢ The tssh alias - Cluster SSH replacement
⇢ ⇢ The tmux::tssh_from_argument helper
⇢ ⇢ The tmux::tssh_from_file helper
⇢ ⇢ tssh examples
⇢ ⇢ Common Tmux commands I use in tssh
⇢ Copy and paste workflow
⇢ Tmux configurations

Before continuing...

-
-Before continuing to read this post, I encourage you to get familiar with Tmux first (unless you already know the basics). You can go through the official getting started guide:
-
-https://github.com/tmux/tmux/wiki/Getting-Started
-
-I can also recommend this book (this is the book I got started with with Tmux):
-
-https://pragprog.com/titles/bhtmux2/tmux-2/
-
-Over the years, I have built a couple of shell helper functions to optimize my workflows. Tmux is extensively integrated into my daily workflows (personal and work). I had colleagues asking me about my Tmux config and helper scripts for Tmux several times. It would be neat to blog about it so that everyone interested in it can make a copy of my configuration and scripts.
-
-The configuration and scripts in this blog post are only the non-work-specific parts. There are more helper scripts, which I only use for work (and aren't really useful outside of work due to the way servers and clusters are structured there).
-
-Tmux is highly configurable, and I think I am only scratching the surface of what is possible with it. Nevertheless, it may still be useful for you. I also love that Tmux is part of the OpenBSD base system!
-
-

Shell aliases

-
-I am a user of the Z-Shell (zsh), but I believe all the snippets mentioned in this blog post also work with Bash.
-
-https://www.zsh.org
-
-For the most common Tmux commands I use, I have created the following shell aliases:
-
- -

alias tm=tmux
-alias tl='tmux list-sessions'
-alias tn=tmux::new
-alias ta=tmux::attach
-alias tx=tmux::remote
-alias ts=tmux::search
-alias tssh=tmux::cluster_ssh
-

-
-Note all tmux::...; those are custom shell functions doing certain things, and they aren't part of the Tmux distribution. But let's run through every aliases one by one.
-
-The first two are pretty straightforward. tm is simply a shorthand for tmux, so I have to type less, and tl lists all Tmux sessions that are currently open. No magic here.
-
-

The tn alias - Creating a new session

-
-The tn alias is referencing this function:
-
- -

# Create new session and if alread exists attach to it
-tmux::new () {
-    readonly session=$1
-    local date=date
-    if where gdate &>/dev/null; then
-        date=gdate
-    fi
-
-    tmux::cleanup_default
-    if [ -z "$session" ]; then
-        tmux::new T$($date +%s)
-    else
-        tmux new-session -d -s $session
-        tmux -2 attach-session -t $session || tmux -2 switch-client -t $session
-    fi
-}
-alias tn=tmux::new
-

-
-There is a lot going on here. Let's have a detailed look at what it is doing. As a note, the function relies on GNU Date, so MacOS is looking for the gdate commands to be available. Otherwise, it will fall back to date. You need to install GNU Date for Mac, as it isn't installed by default there. As I use Fedora Linux on my personal Laptop and a MacBook for work, I have to make it work for both.
-
-First, a Tmux session name can be passed to the function as a first argument. That session name is only optional. Without it, Tmux will select a session named T$($date +%s) as a default. Which is T followed by the UNIX epoch, e.g. T1717133796.
-
-

Cleaning up default sessions automatically

-
-Note also the call to tmux::cleanup_default; it would clean up all already opened default sessions if they aren't attached. Those sessions were only temporary, and I had too many flying around after a while. So, I decided to auto-delete the sessions if they weren't attached. If I want to keep sessions around, I will rename them with the Tmux command prefix-key $. This is the cleanup function:
-
- -

tmux::cleanup_default () {
-    local s
-    tmux list-sessions | grep '^T.*: ' | grep -F -v attached |
-    cut -d: -f1 | while read -r s; do
-        echo "Killing $s"
-        tmux kill-session -t "$s"
-    done
-}
-

-
-The cleanup function kills all open Tmux sessions that haven't been renamed properly yet—but only if they aren't attached (e.g., don't run in the foreground in any terminal). Cleaning them up automatically keeps my Tmux sessions as neat and tidy as possible.
-
-

Renaming sessions

-
-Whenever I am in a temporary session (named T....), I may decide that I want to keep this session around. I have to rename the session to prevent the cleanup function from doing its thing. That's, as mentioned already, easily accomplished with the standard prefix-key $ Tmux command.
-
-

The ta alias - Attaching to a session

-
-This alias refers to the following function, which tries to attach to an already-running Tmux session.
-
- -

tmux::attach () {
-    readonly session=$1
-
-    if [ -z "$session" ]; then
-        tmux attach-session || tmux::new
-    else
-        tmux attach-session -t $session || tmux::new $session
-    fi
-}
-alias ta=tmux::attach
-

-
-If no session is specified (as the argument of the function), it will try to attach to the first open session. If no Tmux server is running, it will create a new one with tmux::new. Otherwise, with a session name given as the argument, it will attach to it. If unsuccessful (e.g., the session doesn't exist), it will be created and attached to.
-
-

The tr alias - For a nested remote session

-
-This SSHs into the remote server specified and then, remotely on the server itself, starts a nested Tmux session. So we have one Tmux session on the local computer and, inside of it, an SSH connection to a remote server with a Tmux session running again. The benefit of this is that, in case my network connection breaks down, the next time I connect, I can continue my work on the remote server exactly where I left off. The session name is the name of the server being SSHed into. If a session like this already exists, it simply attaches to it.
-
- -

tmux::remote () {
-    readonly server=$1
-    tmux new -s $server "ssh -t $server 'tmux attach-session || tmux'" || \
-        tmux attach-session -d -t $server
-}
-alias tr=tmux::remote
-

-
-

Change of the Tmux prefix for better nesting

-
-To make nested Tmux sessions work smoothly, one must change the Tmux prefix key locally or remotely. By default, the Tmux prefix key is Ctrl-b, so Ctrl-b $, for example, renames the current session. To change the prefix key from the standard Ctrl-b to, for example, Ctrl-g, you must add this to the tmux.conf:
-
-

-set-option -g prefix C-g
-

-
-This way, when I want to rename the remote Tmux session, I have to use Ctrl-g $, and when I want to rename the local Tmux session, I still have to use Ctrl-b $. In my case, I have this deployed to all remote servers through a configuration management system (out of scope for this blog post).
-
-There might also be another way around this (without reconfiguring the prefix key), but that is cumbersome to use, as far as I remember.
-
-

The ts alias - Searching sessions with fuzzy finder

-
-Despite the fact that with tmux::cleanup_default, I don't leave a huge mess with trillions of Tmux sessions flying around all the time, at times, it can become challenging to find exactly the session I am currently interested in. After a busy workday, I often end up with around twenty sessions on my laptop. This is where fuzzy searching for session names comes in handy, as I often don't remember the exact session names.
-
- -

tmux::search () {
-    local -r session=$(tmux list-sessions | fzf | cut -d: -f1)
-    if [ -z "$TMUX" ]; then
-        tmux attach-session -t $session
-    else
-        tmux switch -t $session
-    fi
-}
-alias ts=tmux::search
-

-
-All it does is list all currently open sessions in fzf, where one of them can be searched and selected through fuzzy find, and then either switch (if already inside a session) to the other session or attach to the other session (if not yet in Tmux).
-
-You must install the fzf command on your computer for this to work. This is how it looks like:
-
-
-
-

The tssh alias - Cluster SSH replacement

-
-Before I used Tmux, I was a heavy user of ClusterSSH, which allowed me to log in to multiple servers at once in a single terminal window and type and run commands on all of them in parallel.
-
-https://github.com/duncs/clusterssh
-
-However, since I started using Tmux, I retired ClusterSSH, as it came with the benefit that Tmux only needs to be run in the terminal, whereas ClusterSSH spawned terminal windows, which aren't easily portable (e.g., from a Linux desktop to macOS). The tmux::cluster_ssh function can have N arguments, where:
-
-

...the first argument will be the session name (see tmux::tssh_from_argument helper function), and all remaining arguments will be server hostnames/FQDNs to connect to simultaneously.
...or, the first argument is a file name, and the file contains a list of hostnames/FQDNs (see tmux::ssh_from_file helper function)

-This is the function definition behind the tssh alias:
-
- -

tmux::cluster_ssh () {
-    if [ -f "$1" ]; then
-        tmux::tssh_from_file $1
-        return
-    fi
-
-    tmux::tssh_from_argument $@
-}
-alias tssh=tmux::cluster_ssh
-

-
-This function is just a wrapper around the more complex tmux::tssh_from_file and tmux::tssh_from_argument functions, as you have learned already. Most of the magic happens there.
-
-

The tmux::tssh_from_argument helper

-
-This is the most magic helper function we will cover in this post. It looks like this:
-
- -

tmux::tssh_from_argument () {
-    local -r session=$1; shift
-    local first_server=$1; shift
-
-    tmux new-session -d -s $session "ssh -t $first_server"
-    if ! tmux list-session | grep "^$session:"; then
-        echo "Could not create session $session"
-        return 2
-    fi
-
-    for server in "${@[@]}"; do
-        tmux split-window -t $session "tmux select-layout tiled; ssh -t $server"
-    done
-
-    tmux setw -t $session synchronize-panes on
-    tmux -2 attach-session -t $session | tmux -2 switch-client -t $session
-}
-

-
-It expects at least two arguments. The first argument is the session name to create for the clustered SSH session. All other arguments are server hostnames or FQDNs to which to connect. The first one is used to make the initial session. All remaining ones are added to that session with tmux split-window -t $session.... At the end, we enable synchronized panes by default, so whenever you type, the commands will be sent to every SSH connection, thus allowing the neat ClusterSSH feature to run commands on multiple servers simultaneously. Once done, we attach (or switch, if already in Tmux) to it.
-
-Sometimes, I don't want the synchronized panes behavior and want to switch it off temporarily. I can do that with prefix-key p and prefix-key P after adding the following to my local tmux.conf:
-
-

-bind-key p setw synchronize-panes off
-bind-key P setw synchronize-panes on
-

-
-

The tmux::tssh_from_file helper

-
-This one sets the session name to the file name and then reads a list of servers from that file, passing the list of servers to tmux::tssh_from_argument as the arguments. So, this is a neat little wrapper that also enables me to open clustered SSH sessions from an input file.
-
- -

tmux::tssh_from_file () {
-    local -r serverlist=$1; shift
-    local -r session=$(basename $serverlist | cut -d. -f1)
-
-    tmux::tssh_from_argument $session $(awk '{ print $1} ' $serverlist | sed 's/.lan./.lan/g')
-}
-

-
-

tssh examples

-
-To open a new session named fish and log in to 4 remote hosts, run this command (Note that it is also possible to specify the remote user):
-
-

-$ tssh fish blowfish.buetow.org fishfinger.buetow.org \
-    fishbone.buetow.org user@octopus.buetow.org
-

-
-To open a new session named manyservers, put many servers (one FQDN per line) into a file called manyservers.txt and simply run:
-
-

-$ tssh manyservers.txt
-

-
-

Common Tmux commands I use in tssh

-
-These are default Tmux commands that I make heavy use of in a tssh session:
-
-

Press prefix-key DIRECTION to switch panes. DIRECTION is by default any of the arrow keys, but I also configured Vi keybindings.
Press prefix-key to change the pane layout (can be pressed multiple times to cycle through them).
Press prefix-key z to zoom in and out of the current active pane.

Copy and paste workflow

-
-As you will see later in this blog post, I have configured a history limit of 1 million items in Tmux so that I can scroll back quite far. One main workflow of mine is to search for text in the Tmux history, select and copy it, and then switch to another window or session and paste it there (e.g., into my text editor to do something with it).
-
-This works by pressing prefix-key [ to enter Tmux copy mode. From there, I can browse the Tmux history of the current window using either the arrow keys or vi-like navigation (see vi configuration later in this blog post) and the Pg-Dn and Pg-Up keys.
-
-I often search the history backwards with prefix-key [ followed by a ?, which opens the Tmux history search prompt.
-
-Once I have identified the terminal text to be copied, I enter visual select mode with v, highlight all the text to be copied (using arrow keys or Vi motions), and press y to yank it (sorry if this all sounds a bit complicated, but Vim/NeoVim users will know this, as it is pretty much how you do it there as well).
-
-For v and y to work, the following has to be added to the Tmux configuration file:
-
-

-bind-key -T copy-mode-vi 'v' send -X begin-selection
-bind-key -T copy-mode-vi 'y' send -X copy-selection-and-cancel
-

-
-Once the text is yanked, I switch to another Tmux window or session where, for example, a text editor is running and paste the yanked text from Tmux into the editor with prefix-key ]. Note that when pasting into a modal text editor like Vi or Helix, you would first need to enter insert mode before prefix-key ] would paste anything.
-
-

Tmux configurations

-
-Some features I have configured directly in Tmux don't require an external shell alias to function correctly. Let's walk line by line through my local ~/.config/tmux/tmux.conf:
-
-

-source ~/.config/tmux/tmux.local.conf
-
-set-option -g allow-rename off
-set-option -g history-limit 100000
-set-option -g status-bg '#444444'
-set-option -g status-fg '#ffa500'
-set-option -s escape-time 0
-

-
-There's yet to be much magic happening here. I source a tmux.local.conf, which I sometimes use to override the default configuration that comes from the configuration management system. But it is mostly just an empty file, so it doesn't throw any errors on Tmux startup when I don't use it.
-
-I work with many terminal outputs, which I also like to search within Tmux. So, I added a large enough history-limit, enabling me to search backwards in Tmux for any output up to a million lines of text.
-
-Besides changing some colours (personal taste), I also set escape-time to 0, which is just a workaround. Otherwise, my Helix text editor's ESC key would take ages to trigger within Tmux. I am trying to remember the gory details. You can leave it out; if everything works fine for you, leave it out.
-
-The next lines in the configuration file are:
-
-

-set-window-option -g mode-keys vi
-bind-key -T copy-mode-vi 'v' send -X begin-selection
-bind-key -T copy-mode-vi 'y' send -X copy-selection-and-cancel
-

-
-I navigate within Tmux using Vi keybindings, so the mode-keys is set to vi. I use the Helix modal text editor, which is close enough to Vi bindings for simple navigation to feel "native" to me. (By the way, I have been a long-time Vim and NeoVim user, but I eventually switched to Helix. It's off-topic here, but it may be worth another blog post once.)
-
-The two bind-key commands make it so that I can use v and y in copy mode, which feels more Vi-like (as already discussed earlier in this post).
-
-The next set of lines in the configuration file are:
-
-

-bind-key h select-pane -L
-bind-key j select-pane -D
-bind-key k select-pane -U
-bind-key l select-pane -R
-
-bind-key H resize-pane -L 5
-bind-key J resize-pane -D 5
-bind-key K resize-pane -U 5
-bind-key L resize-pane -R 5
-

-
-These allow me to use prefix-key h, prefix-key j, prefix-key k, and prefix-key l for switching panes and prefix-key H, prefix-key J, prefix-key K, and prefix-key L for resizing the panes. If you don't know Vi/Vim/NeoVim, the letters hjkl are commonly used there for left, down, up, and right, which is also the same for Helix, by the way.
-
-The next set of lines in the configuration file are:
-
-

-bind-key c new-window -c '#{pane_current_path}'
-bind-key F new-window -n "session-switcher" "tmux list-sessions | fzf | cut -d: -f1 | xargs tmux switch-client -t"
-bind-key T choose-tree
-

-
-The first one is that any new window starts in the current directory. The second one is more interesting. I list all open sessions in the fuzzy finder. I rely heavily on this during my daily workflow to switch between various sessions depending on the task. E.g. from a remote cluster SSH session to a local code editor.
-
-The third one, choose-tree, opens a tree view in Tmux listing all sessions and windows. This one is handy to get a better overview of what is currently running in any local Tmux session. It looks like this (it also allows me to press a hotkey to switch to a particular Tmux window):
-
-
-
-
-The last remaining lines in my configuration file are:
-
-

-bind-key p setw synchronize-panes off
-bind-key P setw synchronize-panes on
-bind-key r source-file ~/.config/tmux/tmux.conf \; display-message "tmux.conf reloaded"
-

-
-We discussed synchronized panes earlier. I use it all the time in clustered SSH sessions. When enabled, all panes (remote SSH sessions) receive the same keystrokes. This is very useful when you want to run the same commands on many servers at once, such as navigating to a common directory, restarting a couple of services at once, or running tools like htop to quickly monitor system resources.
-
-The last one reloads my Tmux configuration on the fly.
-
-E-Mail your comments to paul@nospam.buetow.org :-)
-
-Back to the main site
-

Projects I currently don't have time for

2024-05-03T16:23:03+03:00

Projects I currently don't have time for

-
-Published at 2024-05-03T16:23:03+03:00
-
-Over the years, I have collected many ideas for my personal projects and noted them down. I am currently in the process of cleaning up all my notes and reviewing those ideas. I don’t have time for the ones listed here and won’t have any soon due to other commitments and personal projects. So, in order to "get rid of them" from my notes folder, I decided to simply put them in this blog post so that those ideas don't get lost. Maybe I will pick up one or another idea someday in the future, but for now, they are all put on ice in favor of other personal projects or family time.
-
-

-Art by Laura Brown
-
-.'`~~~~~~~~~~~`'.
-(  .'11 12 1'.  )
-|  :10 \    2:  |
-|  :9   @-> 3:  |
-|  :8       4;  |
-'. '..7 6 5..' .'
- ~-------------~  ldb
-
-

-
-

Projects I currently don't have time for
⇢ Hardware projects I don't have time for
⇢ ⇢ I use Arch, btw!
⇢ ⇢ OpenBSD home router
⇢ ⇢ Pi-Hole server
⇢ ⇢ Infodash
⇢ ⇢ Reading station
⇢ ⇢ Retro station
⇢ ⇢ Sound server
⇢ ⇢ Project Freekat
⇢ Programming projects I don't have time for
⇢ ⇢ CLI-HIVE
⇢ ⇢ Enhanced KISS home photo albums
⇢ ⇢ KISS file sync server with end-to-end encryption
⇢ ⇢ A language that compiles to bash
⇢ ⇢ A language that compiles to sed
⇢ ⇢ Renovate VS-Sim
⇢ ⇢ KISS ticketing system
⇢ ⇢ A domain-specific language (DSL) for work
⇢ Self-hosting projects I don't have time for
⇢ ⇢ My own Matrix server
⇢ ⇢ Ampache music server
⇢ ⇢ Librum eBook reader
⇢ ⇢ Memos - Note-taking service
⇢ ⇢ Bepasty server
⇢ Books I don't have time to read
⇢ ⇢ Fluent Python
⇢ ⇢ Programming Ruby
⇢ ⇢ Peter F. Hamilton science fiction books
⇢ New websites I don't have time for
⇢ ⇢ Create a "Why Raku Rox" site
⇢ Research projects I don't have time for
⇢ ⇢ Project secure
⇢ ⇢ CPU utilisation is all wrong

Hardware projects I don't have time for

-
-

I use Arch, btw!

-
-The idea was to build the ultimate Arch Linux setup on an old ThinkPad X200 booting with the open-source LibreBoot firmware, complete with a tiling window manager, dmenu, and all the elite tools. This is mainly for fun, as I am pretty happy (and productive) with my Fedora Linux setup. I ran EndeavourOS (close enough to Arch) on an old ThinkPad for a while, but then I switched back to Fedora because the rolling releases were annoying (there were too many updates).
-
-

OpenBSD home router

-
-In my student days, I operated a 486DX PC with OpenBSD as my home DSL internet router. I bought the setup from my brother back then. The router's hostname was fishbone, and it performed very well until it became too slow for larger broadband bandwidth after a few years of use.
-
-I had the idea to revive this concept, implement fishbone2, and place it in front of my proprietary ISP router to add an extra layer of security and control in my home LAN. It would serve as the default gateway for all of my devices, including a Wi-Fi access point, would run a DNS server, Pi-hole proxy, VPN client, and DynDNS client. I would also implement high availability using OpenBSD's CARP protocol.
-
-https://openbsdrouterguide.net
-https://pi-hole.net/
-https://www.OpenBSD.org
-https://www.OpenBSD.org/faq/pf/carp.html
-
-However, I am putting this on hold as I have opted for an OpenWRT-based solution, which was much quicker to set up and runs well enough.
-
-https://OpenWRT.org/
-
-

Pi-Hole server

-
-Install Pi-hole on one of my Pis or run it in a container on Freekat. For now, I am putting this on hold as the primary use for this would be ad-blocking, and I am avoiding surfing ad-heavy sites anyway. So there's no significant use for me personally at the moment.
-
-https://pi-hole.net/
-
-

Infodash

-
-The idea was to implement my smart info screen using purely open-source software. It would display information such as the health status of my personal infrastructure, my current work tracker balance (I track how much I work to prevent overworking), and my sports balance (I track my workouts to stay within my quotas for general health). The information would be displayed on a small screen in my home office, on my Pine watch, or remotely from any terminal window.
-
-I don't have this, and I haven't missed having it, so I guess it would have been nice to have it but not provide any value other than the "fun of tinkering."
-
-

Reading station

-
-I wanted to create the most comfortable setup possible for reading digital notes, articles, and books. This would include a comfy armchair, a silent barebone PC or Raspberry Pi computer running either Linux or *BSD, and an e-Ink display mounted on a flexible arm/stand. There would also be a small table for my paper journal for occasional note-taking. There are a bunch of open-source software available for PDF and ePub reading. It would have been neat, but I am currently using the most straightforward solution: a Kobo Elipsa 2E, which I can use on my sofa.
-
-

Retro station

-
-I had an idea to build a computer infused with retro elements. It wouldn't use actual retro hardware but would look and feel like a retro machine. I would call this machine HAL or Retron.
-
-I would use an old ThinkPad laptop placed on a horizontal stand, running NetBSD, and attaching a keyboard from ModelFkeyboards. I use WindowMaker as a window manager and run terminal applications through Retro Term. For the monitor, I would use an older (black) EIZO model with large bezels.
-
-https://www.NetBSD.org
-https://www.modelfkeyboards.com
-https://github.com/Swordfish90/cool-retro-term)
-
-The computer would occasionally be used to surf the Gemini space, take notes, blog, or do light coding. However, I have abandoned the project for now because there isn't enough space in my apartment, as my daughter will have a room for herself.
-
-

Sound server

-
-My idea involved using a barebone mini PC running FreeBSD with the Navidrome sound server software. I could remotely connect to it from my phone, workstation/laptop to listen to my music collection. The storage would be based on ZFS with at least two drives for redundancy. The app would run in a Linux Docker container under FreeBSD via Bhyve.
-
-https://github.com/navidrome/navidrome
-https://wiki.freebsd.org/bhyve
-
-

Project Freekat

-
-My idea involved purchasing the Meerkat mini PC from System76 and installing FreeBSD. Like the sound-server idea (see previous idea), it would run Linux Docker through Bhyve. I would self-host a bunch of applications on it:
-
-

Wallabag
Ankidroid
Miniflux & Postgres
Audiobookshelf
...

-All of this would be within my LAN, but the services would also be accessible from the internet through either Wireguard or SSH reverse tunnels to one of my OpenBSD VMs, for example:
-
-

wallabag.awesome.buetow.org
ankidroid.awesome.buetow.org
miniflux.awesome.buetow.org
audiobookshelf.awesome.buetow.org
...

-I am abandoning this project for now, as I am currently hosting my apps on AWS ECS Fargate under *.cool.buetow.org, which is "good enough" for the time being and also offers the benefit of learning to use AWS and Terraform, knowledge that can be applied at work.
-
-My personal AWS setup
-
-

Programming projects I don't have time for

-
-

CLI-HIVE

-
-This was a pet project idea that my brother and I had. The concept was to collect all shell history of all servers at work in a central place, apply ML/AI, and return suggestions for commands to type or allow a fuzzy search on all the commands in the history. The recommendations for the commands on a server could be context-based (e.g., past occurrences on the same server type).
-
-You could decide whether to share your command history with others so they would receive better suggestions depending on which server they are on, or you could keep all the history private and secure. The plan was to add hooks into zsh and bash shells so that all commands typed would be pushed to the central location for data mining.
-
-

Enhanced KISS home photo albums

-
-I don't use third-party cloud providers such as Google Photos to store/archive my photos. Instead, they are all on a ZFS volume on my home NAS, with regular offsite backups taken. Thus, my project would involve implementing the features I miss most or finding a solution simple enough to host on my LAN:
-
-

A feature I miss presents me with a random day from the past and some photos from that day. This project would randomly select a day and generate a photo album for me to view and reminisce about memories.
Another feature I miss is the ability to automatically deduplicate all the photos, as I am sure there are tons of duplicates on my NAS.
Auto-enhancing the photos (perhaps using ImageMagick?)
I already have a simple photoalbum.sh script that generates an album based on an input directory. However, it would be great also to have a timeline feature to enable browsing through different dates.

-KISS static web photo albums with photoalbum.sh
-
-

KISS file sync server with end-to-end encryption

-
-I aimed to have a simple server to which I could sync notes and other documents, ensuring that the data is fully end-to-end encrypted. This way, only the clients could decrypt the data, while an encrypted copy of all the data would be stored on the server side. There are a few solutions (e.g., NextCloud), but they are bloated or complex to set up.
-
-I currently use Syncthing for encrypted file sync across all my devices; however, the data is not end-to-end encrypted. It's a good-enough setup, though, as my Syncthing server is in my home LAN on an encrypted file system.
-
-https://syncthing.net
-
-I also had the idea of using this as a pet project for work and naming it Cryptolake, utilizing post-quantum-safe encryption algorithms and a distributed data store.
-
-

A language that compiles to bash

-
-I had an idea to implement a higher-level language with strong typing that could be compiled into native Bash code. This would make all resulting Bash scripts more robust and secure by default. The project would involve developing a parser, lexer, and a Bash code generator. I planned to implement this in Go.
-
-I had previously implemented a tiny scripting language called Fype (For Your Program Execution), which could have served as inspiration.
-
-The Fype Programming Language
-
-

A language that compiles to sed

-
-This is similar to the previous idea, but the difference is that the language would compile into a sed script. Sed has many features, but the brief syntax makes scripts challenging to read. The higher-level language would mimic sed but in a form that is easier for humans to read.
-
-

Renovate VS-Sim

-
-VS-Sim is an open-source simulator programmed in Java for distributed systems. VS-Sim stands for "Verteilte Systeme Simulator," the German translation for "Distributed Systems Simulator." The VS-Sim project was my diploma thesis at Aachen University of Applied Sciences.
-
-https://codeberg.org/snonux/vs-sim
-
-The ideas I had was:
-
-

Translate the project into English.
Modernise the Java codebase to be compatible with the latest JDK.
Make it compile to native binaries using GraalVM.
Distribute the project using AppImages.

-I have put this project on hold for now, as I want to do more things in Go and fewer in Java in my personal time.
-
-

KISS ticketing system

-
-My idea was to program a KISS (Keep It Simple, Stupid) ticketing system for my personal use. However, I am abandoning this project because I now use the excellent Taskwarrior software. You can learn more about it at:
-
-https://taskwarrior.org/
-
-

A domain-specific language (DSL) for work

-
-At work, an internal service allocates storage space for our customers on our storage clusters. It automates many tasks, but many tweaks are accessible through APIs. I had the idea to implement a Ruby-based DSL that would make using all those APIs for ad-hoc changes effortless, e.g.:
-
- -

Cluster :UK, :uk01 do
-  Customer.C1A1.segments.volumes.each do |volume|
-    puts volume.usage_stats
-    volume.move_off! if volume.over_subscribed?
-  end
-end
-

-
-I am abandoning this project because my workplace has stopped the annual pet project competition, and I have other more important projects to work on at the moment.
-
-Creative universe (Work pet project contests)
-
-

Self-hosting projects I don't have time for

-
-

My own Matrix server

-
-I value privacy. It would be great to run my own Matrix server for communication within my family. I have yet to have time to look into this more closely.
-
-https://matrix.org
-
-

Ampache music server

-
-Ampache is an open-source music streaming server that allows you to host and manage your music collection online, accessible via a web interface. Setting it up involves configuring a web server, installing Ampache, and organising your music files, which can be time-consuming.
-
-

Librum eBook reader

-
-Librum is a self-hostable e-book reader that allows users to manage and read their e-book collection from a web interface. Designed to be a self-contained platform where users can upload, organise, and access their e-books, Librum emphasises privacy and control over one's digital library.
-
-https://github.com/Librum-Reader/Librum
-
-I am using my Kobo devices or my laptop to read these kinds of things for now.
-
-

Memos - Note-taking service

-
-Memos is a note-taking service that simplifies and streamlines information capture and organisation. It focuses on providing users with a minimalistic and intuitive interface, aiming to enhance productivity without the clutter commonly associated with more complex note-taking apps.
-
-https://www.usememos.com
-
-I am abandoning this idea for now, as I am currently using plain Markdown files for notes and syncing them with Syncthing across my devices.
-
-

Bepasty server

-
-Bepasty is like a Pastebin for all kinds of files (text, image, audio, video, documents, binary, etc.). It seems very neat, but I only share a little nowadays. When I do, I upload files via SCP to one of my OpenBSD VMs and serve them via vanilla httpd there, keeping it KISS.
-
-https://github.com/bepasty/bepasty-server
-
-

Books I don't have time to read

-
-

Fluent Python

-
-I consider myself an advanced programmer in Ruby, Bash, and Perl. However, Python seems to be ubiquitous nowadays, and most of my colleagues prefer Python over any other languages. Thus, it makes sense for me to also learn and use Python. After conducting some research, "Fluent Python" appears to be the best book for this purpose.
-
-I don't have time to read this book at the moment, as I am focusing more on Go (Golang) and I know just enough Python to get by (e.g., for code reviews). Additionally, there are still enough colleagues around who can review my Ruby or Bash code.
-
-

Programming Ruby

-
-I've read a couple of Ruby books already, but "Programming Ruby," which covers up to Ruby 3.2, was just recently released. I would like to read this to deepen my Ruby knowledge further and to revisit some concepts that I may have forgotten.
-
-As stated in this blog post, I am currently more eager to focus on Go, so I've put the Ruby book on hold. Additionally, there wouldn't be enough colleagues who could "understand" my advanced Ruby skills anyway, as most of them are either Java developers or SREs who don't code a lot.
-
-

Peter F. Hamilton science fiction books

-
-I am a big fan of science fiction, but my reading list is currently too long anyway. So, I've put the Hamilton books on the back burner for now. You can see all the novels I've read here:
-
-https://paul.buetow.org/novels.html
-https://paul.buetow.org/novels.gmi
-
-
-

New websites I don't have time for

-
-

Create a "Why Raku Rox" site

-
-The website "Why Raku Rox" would showcase the unique features and benefits of the Raku programming language and highlight why it is an exceptional choice for developers. Raku, originally known as Perl 6, is a dynamic, expressive language designed for flexible and powerful software development.
-
-This would be similar to the "Why OpenBSD rocks" site:
-
-https://why-openbsd.rocks
-https://raku.org
-
-I am not working on this for now, as I currently don’t even have time to program in Raku.
-
-

Research projects I don't have time for

-
-

Project secure

-
-For work: Implement a PoC that dumps Java heaps to extract secrets from memory. Based on the findings, write a Java program that encrypts secrets in the kernel using the memfd_secret() syscall to make it even more secure.
-
-https://lwn.net/Articles/865256/
-
-Due to other priorities, I am putting this on hold for now. The software we have built is pretty damn secure already!
-
-

CPU utilisation is all wrong

-
-This research project, based on Brendan Gregg's blog post, could potentially significantly impact my work.
-
-https://brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html
-
-The research project would involve setting up dashboards that display actual CPU usage and the cycles versus waiting time for memory access.
-
-E-Mail your comments to paul@nospam.buetow.org :-)
-
-Related and maybe interesting:
-
-Sweating the small stuff - Tiny projects of mine
-
-Back to the main site
-

'Slow Productivity' book notes

2024-04-27T14:18:51+03:00

"Slow Productivity" book notes

-
-Published at 2024-04-27T14:18:51+03:00
-
-These are my personal takeaways after reading "Slow Productivity - The lost Art of Accomplishment Without Burnout" by Cal Newport.
-
-The case studies in this book were a bit long, but they appeared to be well-researched. I will only highlight the interesting, actionable items in the book notes.
-
-These notes are mainly for my own use, but you may find them helpful.
-
-

-         ,..........   ..........,
-     ,..,'          '.'          ',..,
-    ,' ,'            :            ', ',
-   ,' ,'             :             ', ',
-  ,' ,'              :              ', ',
- ,' ,'............., : ,.............', ',
-,'  '............   '.'   ............'  ',
- '''''''''''''''''';''';''''''''''''''''''
-                    '''
-

-
-

"Slow Productivity" book notes
⇢ It's not "slow productivity"
⇢ Pseudo-productivity and Shallow work
⇢ Accomplishments without burnout
⇢ Do fewer things
⇢ Work at a natural pace
⇢ Obsess over quality

It's not "slow productivity"

-
-"Slow productivity" does not mean being less productive. Cal Newport wants to point out that you can be much more productive with "slow productivity" than you would be without it. It is a different way of working than most of us are used to in the modern workplace, which is hyper-connected and always online.
-
-

Pseudo-productivity and Shallow work

-
-People use visible activity instead of real productivity because it's easier to measure. This is called pseudo-productivity.
-Pseudo-productivity is used as a proxy for real productivity. If you don't look busy, you are dismissed as lazy or lacking a work ethic.
-
-There is a tendency to perform shallow work because people will otherwise dismiss you as lazy. A lot of shallow work can cause burnout, as multiple things are often being worked on in parallel. The more you have on your plate, the more stressed you will be.
-
-Shallow work usually doesn't help you to accomplish big things. Always have the big picture in mind. Shallow work can't be entirely eliminated, but it can be managed—for example, plan dedicated time slots for certain types of shallow work.
-
-

Accomplishments without burnout

-
-The overall perception is that if you want to accomplish something, you must put yourself on the verge of burnout. Cal Newport writes about "The lost Art of Accomplishments without Burnouts", where you can accomplish big things without all the stress usually involved.
-
-There are three principles for the maintenance of a sustainable work life:
-
-

Do fewer things
Work at a natural pace
Obsess over quality

Do fewer things

-
-There will always be more work. The faster you finish it, the quicker you will have something new on your plate.
-
-Reduce the overhead tax. The overhead tax is all the administrative work to be done. With every additional project, there will also be more administrative stuff to be done on your work plate. So, doing fewer things leads to more and better output and better quality for the projects you are working on.
-
-Limit the things on your plate. Limit your missions (personal goals, professional goals). Reduce your main objectives in life. More than five missions are usually not sustainable very easily, so you have to really prioritise what is important to you and your professional life.
-
-A mission is an overall objective/goal that can have multiple projects. Limit the projects as well. Some projects need clear endings (e.g., work in support of a never-ending flow of incoming requests). In this case, set limits (e.g., time box your support hours). You can also plan "office hours" for collaborative work with colleagues to avoid ad hoc distractions.
-
-The key point is that after making these commitments, you really deliver on them. This builds trust, and people will leave you alone and not ask for progress all the time.
-
-Doing fever things is essential for modern knowledge workers. Breathing space in your work also makes you more creative and happier overall.
-
-Pushing workers more work can make them less productive, so the better approach is the pull model, where workers pull in new work when the previous task is finished.
-
-If you can quantify how busy you are or how many other projects you already work on, then it is easier to say no to new things. For example, show what you are doing, what's in the roadmap, etc. Transparency is the key here.
-
-You can have your own simulated pull system if the company doesn't agree to a global one:
-
-

State which additional information you would need.
Create a rough estimate of when you will be able to work on it
Estimate how long the project would take. Double that estimate, as humans are very bad estimators.
Respond to the requester and state that you will let him know when the estimates change.

-Sometimes, a little friction is all that is needed to combat incoming work, e.g., when your manager starts seeing the reality of your work plate, and you also request additional information for the task. If you already have too much on your plate, then decline the new project or make room for it in your calendar. If you present a large task list, others will struggle to assign more to you.
-
-Limit your daily goals. A good measure is to focus on one goal per day. You can time block time for deep work on your daily goal. During that time, you won't be easily available to others.
-
-The battle against distractions must be fought to be the master of your time. Nobody will fight this war for you. You have to do it for yourself. (Also, have a look at Cal Newport's "time block planning" method).
-
-Put tasks on autopilot (regular recurring tasks).
-
-

Work at a natural pace

-
-We suffer from overambitious timelines, task lists, and business. Focus on what matters. Don't rush your most important work to achieve better results.
-
-Don't rush. If you rush or are under pressure, you will be less effective and eventually burn out. Our brains work better then not rushy. The stress heuristic usually indicates too much work, and it is generally too late to reduce workload. That's why we all typically have dangerously too much to do.
-
-Have the courage to take longer to do things that are important. For example, plan on a yearly and larger scale, like 2 to 5 years.
-
-Find a reasonable time for a project and then double the project timeline against overconfident optimism. Humans are not great at estimating. They gravitate towards best-case estimates. If you have planned more than enough time for your project, then you will fall into a natural work pace. Otherwise, you will struggle with rushing and stress.
-
-Some days will still be intense and stressful, but those are exceptional cases. After those exceptions (e.g., finalizing that thing, etc.), calmer periods will follow again.
-
-Pace yourself over modest results over time. Simplify and reduce the daily task lists. Meetings: Certain hours are protected for work. For each meeting, add a protected block to your calendar, so you attend meetings only half a day max.
-
-Schedule slow seasons (e.g., when on vacation). Disconnect in the slow season. Doing nothing will not satisfy your mind, though. You could read a book on your subject matter to counteract that.
-
-

Obsess over quality

-
-Obsess over quality even if you lose short-term opportunities by rejecting other projects. Quality demands you slow down. The two previous two principles (do fewer things and work at a natural pace) are mandatory for this principle to work:
-
-

Focus on the core activities of your work for your obsession - you will only have the time to obsess over some things.
Deliver solid work with good quality.
Sharpen the focus to do the best work possible.

-Go pro to save time, and don't squeeze everything out that you can from freemium services. Professional software services eliminate administrative work:
-
-

Pay people who know what they are doing and focus on your stuff.
For example, don't repair that car if you know the mechanic can do that much better than you.
Or don't use the free version of the music streaming service if it interrupts you with commercials, hindering your ability to concentrate on your work.
Hire an accountant for your yearly tax returns. He knows much more about that stuff than you do. And in the end, he will even be cheaper as he knows all the tax laws.
...

-Adjust your workplace to what you want to accomplish. You could have dedicated places in your home for different things, e.g., a place where you read and think (armchair) and a place where you collaborate (your desk or whiteboard). Surround yourself with things that inspire you (e.g., your favourite books on your shelf next to you, etc.).
-
-There is the concept of quiet quitting. It doesn't mean quitting your job, but it means that you don't go beyond and above the expectations people have of you. Quiet quitting became popular with modern work, which is often meaningless and full of shallow tasks. If you obsess over quality, you enjoy your craft and want to go beyond and above.
-
-Implement rituals and routines which shift you towards your goals:
-
-

For example, if you want to be a good Software Engineer, you also have to put in the work regularly. For instance, progress a bit every day in your project at hand, even if it is only one hour daily. Also, a little quality daily work will be more satisfying over time than many shallow tasks.
Do you want to be lean and/or healthy? Schedule your daily walks and workouts. They will become habits over time.
There's the compounding effect where every small effort made every day will yield significant results in the long run

-Deciding what not to do is as important as deciding what to do.
-
-It appears to be money thrown out of the window, but you get a $50 expensive paper notebook (and also a good pen). Unconsciously, it will make you take notes more seriously. You will think about what to put into the notebooks more profoundly and have thought through the ideas more intensively. If you used very cheap notebooks, you would scribble a lot of rubbish and wouldn't even recognise your handwriting after a while anymore. So choosing a high-quality notebook will help you to take higher-quality notes, too.
-
-Slow productivity is actionable and can be applied immediately.
-
-E-Mail your comments to paul@nospam.buetow.org :-)
-
-Other book notes of mine are:
-
-2025-11-02 "The Courage To Be Disliked" book notes
-2025-06-07 "A Monk's Guide to Happiness" book notes
-2025-04-19 "When: The Scientific Secrets of Perfect Timing" book notes
-2024-10-24 "Staff Engineer" book notes
-2024-07-07 "The Stoic Challenge" book notes
-2024-05-01 "Slow Productivity" book notes (You are currently reading this)
-2023-11-11 "Mind Management" book notes
-2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes
-2023-05-06 "The Obstacle is the Way" book notes
-2023-04-01 "Never split the difference" book notes
-2023-03-16 "The Pragmatic Programmer" book notes
-
-Back to the main site
-

KISS high-availability with OpenBSD

2024-03-30T22:12:56+02:00

KISS high-availability with OpenBSD

-
-Published at 2024-03-30T22:12:56+02:00
-
-I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like (in unsorted and slightly unrelated order) BGP, LVS/IPVS, ldirectord, Pacemaker, STONITH, scripted VIP failover via ARP, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work.
-
-But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space - something as KISS (keep it simple and stupid) as possible.
-
-It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway.
-
-PS: ASCII-art below reflects an OpenBSD under-water world with all the tools available in the base system.
-
-

-Art by Michael J. Penick (mod. by Paul B.)
-                                               ACME-sky
-        __________
-       / nsd tower\                                             (
-      /____________\                                           (\) awk-ward
-       |:_:_:_:_:_|                                             ))   plant
-       |_:_,--.:_:|                       dig-bubble         (\//   )
-       |:_:|__|_:_|  relayd-castle          _               ) ))   ((
-    _  |_   _  :_:|   _   _   _            (_)             ((((   /)\`
-   | |_| |_| |   _|  | |_| |_| |             o              \\)) (( (
-    \_:_:_:_:/|_|_|_|\:_:_:_:_/             .                ((   ))))
-     |_,-._:_:_:_:_:_:_:_.-,_|                                )) ((//
-     |:|_|:_:_:,---,:_:_:|_|:|                               ,-.  )/
-     |_:_:_:_,'puffy `,_:_:_:_|           _  o               ,;'))((
-     |:_:_:_/  _ | _  \_:_:_:|          (_O                   ((  ))
-_____|_:_:_|  (o)-(o)  |_:_:_|--'`-.     ,--. ksh under-water (((\'/
- ', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\  /,---.~  goat     \`))
-.  ` |_:_:_|   \`-'/   |_:_:_|.  ` .  `  /()\.__( ) .,-----'`-\(( sed-root
- ', ;|:_:_:|    `-'    |:_:_:| ', ; ', ; `--'|   \ ', ; ', ; ',')).,--
-.  ` MJP ` .  ` .  ` .  ` . httpd-soil ` .    .  ` .  ` .  ` .  ` .  `
- ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ;
-
-

-
-

KISS high-availability with OpenBSD
⇢ My auto-failover requirements
⇢ My HA solution
⇢ ⇢ Only OpenBSD base installation required
⇢ ⇢ Fairly cheap and geo-redundant
⇢ ⇢ Failover time and split-brain
⇢ ⇢ Failover support for multiple protocols
⇢ ⇢ Let's encrypt TLS certificates
⇢ ⇢ Monitoring
⇢ ⇢ Rex automation
⇢ More HA

My auto-failover requirements

-
-

Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible.
Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month already!).
It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.
It should be geo-redundant.
It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.
Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.
Let's Encrypt TLS certificates should always work (before and after a failover).
Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover.
Don't configure everything manually. The configuration should be automated and reproducible.

My HA solution

-
-

Only OpenBSD base installation required

-
-My HA solution for Web and Gemini is based on DNS (OpenBSD's nsd) and a simple shell script (OpenBSD's ksh and some little sed and awk and grep). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system.
-
-https://man.OpenBSD.org/nsd.8
-https://man.OpenBSD.org/ksh
-https://man.OpenBSD.org/awk
-https://man.OpenBSD.org/sed
-https://man.OpenBSD.org/dig
-https://man.OpenBSD.org/ftp
-https://man.OpenBSD.org/cron
-
-I also used the dig (for DNS checks) and ftp (for HTTP/HTTPS checks) programs.
-
-The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The ksh script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM:
-
- -

#!/bin/ksh
-
-ZONES_DIR=/var/nsd/zones/master/
-DEFAULT_MASTER=fishfinger.buetow.org
-DEFAULT_STANDBY=blowfish.buetow.org
-
-determine_master_and_standby () {
-    local master=$DEFAULT_MASTER
-    local standby=$DEFAULT_STANDBY
-
-    .
-    .
-    .
-    
-    local -i health_ok=1
-    if ! ftp -4 -o - https://$master/index.txt | grep -q "Welcome to $master"; then
-        echo "https://$master/index.txt IPv4 health check failed"
-        health_ok=0
-    elif ! ftp -6 -o - https://$master/index.txt | grep -q "Welcome to $master"; then
-        echo "https://$master/index.txt IPv6 health check failed"
-        health_ok=0
-    fi
-    if [ $health_ok -eq 0 ]; then
-        local tmp=$master
-        master=$standby
-        standby=$tmp
-    fi
-
-    .
-    .
-    .
-}
-

-
-The failover scripts looks for the ; Enable failover string in the DNS zone files and swaps the A and AAAA records of the DNS entries accordingly:
-
- -

fishfinger$ grep failover /var/nsd/zones/master/foo.zone.zone
-        300 IN A 46.23.94.99 ; Enable failover
-        300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover
-www     300 IN A 46.23.94.99 ; Enable failover
-www     300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover
-standby  300 IN A 23.88.35.144 ; Enable failover
-standby  300 IN AAAA 2a01:4f8:c17:20f1::42 ; Enable failover
-

-
- -

transform () {
-  sed -E '
-	/IN A .*; Enable failover/ {
-	    /^standby/! {
-	        s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/master_a)' ; \3/;
-	    }
-	    /^standby/ {
-	        s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/standby_a)' ; \3/;
-	    }
-	}
-	/IN AAAA .*; Enable failover/ {
-	    /^standby/! {
-	        s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/master_aaaa)' ; \3/;
-	    }
-	    /^standby/ {
-	        s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/standby_aaaa)' ; \3/;
-	    }
-	}
-	/ ; serial/ {
-	    s/^( +) ([0-9]+) .*; (.*)/\1 '$(date +%s)' ; \3/;
-	}
-  '
-}
-

-
-After the failover, the script reloads nsd and performs a sanity check to see if DNS still works. If not, a rollback will be performed:
-
- -

#! Race condition !#
-   
-if [ -f $zone_file.bak ]; then
-    mv $zone_file.bak $zone_file
-fi
-
-cat $zone_file | transform > $zone_file.new.tmp 
-
-grep -v ' ; serial' $zone_file.new.tmp > $zone_file.new.noserial.tmp
-grep -v ' ; serial' $zone_file > $zone_file.old.noserial.tmp
-
-echo "Has zone $zone_file changed?"
-if diff -u $zone_file.old.noserial.tmp $zone_file.new.noserial.tmp; then
-    echo "The zone $zone_file hasn't changed"
-    rm $zone_file.*.tmp
-    return 0
-fi
-
-cp $zone_file $zone_file.bak
-mv $zone_file.new.tmp $zone_file
-rm $zone_file.*.tmp
-echo "Reloading nsd"
-nsd-control reload
-
-if ! zone_is_ok $zone; then
-    echo "Rolling back $zone_file changes"
-    cp $zone_file $zone_file.invalid
-    mv $zone_file.bak $zone_file
-    echo "Reloading nsd"
-    nsd-control reload
-    zone_is_ok $zone
-    return 3
-fi
-
-for cleanup in invalid bak; do
-    if [ -f $zone_file.$cleanup ]; then
-        rm $zone_file.$cleanup
-    fi
-done
-
-echo "Failover of zone $zone to $MASTER completed"
-return 1
-

-
-A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output.
-
-The authorative nameserver for my domains runs on both VMs, and both are configured to be a "master" DNS server so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case.
-
-Check out the whole script here:
-
-dns-failover.ksh
-
-

Fairly cheap and geo-redundant

-
-I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany).
-
-https://OpenBSD.Amsterdam
-https://www.Hetzner.cloud
-
-I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required.
-
-A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions.
-
-

Failover time and split-brain

-
-A DNS failover doesn't happen immediately. I've configured a DNS TTL of 300 seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover).
-
-A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated.
-
-

Failover support for multiple protocols

-
-With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's httpd (OpenBSD's HTTP server) and relayd (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entries, which VM receives the requests.
-
-https://man.OpenBSD.org/httpd.8
-https://man.OpenBSD.org/relayd.8
-
-For example, the master is responsible for the https://www.foo.zone and https://foo.zone hosts, whereas the standby can be reached via https://standby.foo.zone (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. irregular.ninja, paul.buetow.org and so on. The same applies to my Gemini capsules for https://foo.zone, https://standby.foo.zone, https://paul.buetow.org and https://standby.paul.buetow.org.
-
-On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)!
-
-

Let's encrypt TLS certificates

-
-All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate.
-
-If the master always serves foo.zone and the standby always standby.foo.zone, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for foo.zone and the new standby wouldn't have a valid certificate for standby.foo.zone which would lead to TLS errors on the clients.
-
-As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that:
-
-

In an odd week number, the first server is the default master
In an even week number, the second server is the default master.

-Which translates to:
-
- -

# Weekly auto-failover for Let's Encrypt automation
-local -i -r week_of_the_year=$(date +%U)
-if [ $(( week_of_the_year % 2 )) -eq 0 ]; then
-    local tmp=$master
-    master=$standby
-    standby=$tmp
-fi
-

-
-This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM.
-
-The ACME automation is yet another daily CRON script /usr/local/bin/acme.sh. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt.
-
-Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty.
-
-acme.sh.tpl - Rex template for the acme.sh script of mine.
-https://man.OpenBSD.org/acme-client.1
-Let's Encrypt with OpenBSD and Rex
-
-

Monitoring

-
-CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed.
-
-https://codeberg.org/snonux/gogios
-KISS server monitoring with Gogios
-
-Gogios, as I developed it by myself, isn't part of the OpenBSD base system.
-
-

Rex automation

-
-I use Rexify, a friendly configuration management system that allows automatic deployment and configuration.
-
-https://www.rexify.org
-codeberg.org/snonux/rexfiles/frontends
-
-Rex isn't part of the OpenBSD base system, but I didn't need to install any external software on OpenBSD either as Rex is invoked from my Laptop!
-
-

More HA

foo.zone feed

Terminal multiplexing with `tmux` - Z-Shell edition

Terminal multiplexing with tmux - Z-Shell edition

Table of Contents

Before continuing...

Shell aliases

The tn alias - Creating a new session

Cleaning up default sessions automatically

Renaming sessions

The ta alias - Attaching to a session

The tr alias - For a nested remote session

Change of the Tmux prefix for better nesting

The ts alias - Searching sessions with fuzzy finder

The tssh alias - Cluster SSH replacement

The tmux::tssh_from_argument helper

The tmux::tssh_from_file helper

tssh examples

Common Tmux commands I use in tssh

Copy and paste workflow

Tmux configurations

Projects I currently don't have time for

Projects I currently don't have time for

Table of Contents

Hardware projects I don't have time for

I use Arch, btw!

OpenBSD home router

Pi-Hole server

Infodash

Reading station

Retro station

Sound server

Project Freekat

Programming projects I don't have time for

CLI-HIVE

Enhanced KISS home photo albums

KISS file sync server with end-to-end encryption

A language that compiles to bash

A language that compiles to sed

Renovate VS-Sim

KISS ticketing system

A domain-specific language (DSL) for work

Self-hosting projects I don't have time for

My own Matrix server

Ampache music server

Librum eBook reader

Memos - Note-taking service

Bepasty server

Books I don't have time to read

Fluent Python

Programming Ruby

Peter F. Hamilton science fiction books

New websites I don't have time for

Create a "Why Raku Rox" site

Research projects I don't have time for

Project secure

CPU utilisation is all wrong

'Slow Productivity' book notes

"Slow Productivity" book notes

Table of Contents

It's not "slow productivity"

Pseudo-productivity and Shallow work

Accomplishments without burnout

Do fewer things

Work at a natural pace

Obsess over quality

KISS high-availability with OpenBSD

KISS high-availability with OpenBSD

Table of Contents

My auto-failover requirements

My HA solution

Only OpenBSD base installation required

Fairly cheap and geo-redundant

Failover time and split-brain

Failover support for multiple protocols

Let's encrypt TLS certificates

Monitoring

Rex automation

More HA