diff options
| -rw-r--r-- | gemfeed/2024-06-23-terminal-multiplexing-with-tmux.md | 405 | ||||
| -rw-r--r-- | gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.md | 286 | ||||
| -rw-r--r-- | gemfeed/index.md | 1 | ||||
| -rw-r--r-- | gemfeed/terminal-multiplexing-with-tmux/tmux-session-fzf.png | bin | 0 -> 34897 bytes | |||
| -rw-r--r-- | gemfeed/terminal-multiplexing-with-tmux/tmux-tree-view.png | bin | 0 -> 56847 bytes | |||
| -rw-r--r-- | index.md | 3 | ||||
| -rw-r--r-- | uptime-stats.md | 2 |
7 files changed, 409 insertions, 288 deletions
diff --git a/gemfeed/2024-06-23-terminal-multiplexing-with-tmux.md b/gemfeed/2024-06-23-terminal-multiplexing-with-tmux.md new file mode 100644 index 00000000..277968e1 --- /dev/null +++ b/gemfeed/2024-06-23-terminal-multiplexing-with-tmux.md @@ -0,0 +1,405 @@ +# Terminal multiplexing with `tmux` + +> Published at 2024-06-23T22:41:59+03:00 + +``` + \\\\\\\ + \\\\\\\\\\\\ + \\\\\\\\\\\\\\\ + -----------,-| |C> // )\\\\| + ,','| / || ,'/////| +---------,',' X| (, || ///// + || U | \\ ||||//''''| + || M .| ||||||| _| + ||T . |______ `````\____/ \ + || . | ,| _/_____/ \ + ||$ ,' ,' | / | + ||,' ,' | | \ | +_________|/ ,' | / | | +_____________,' ,',_____| | | | + | ,',' | | | | + | ,',' ____|_____/ / | + | ,',' __/ | / | +_____________|',' ///_/-------------/ | + |===========,' + +``` + +``` +Table of contents +================= + +Terminal multiplexing with `tmux` + Introduction + Shell aliases + The `tn` alias - Creating a new session + Cleaning up default sessions automatically + Renaming sessions + The `ta` alias - Attaching to a session + The `tr` alias - For a nested remote session + Change of the Tmux prefix for better nesting + The `ts` alias - Searching sessions with fuzzy finder + The `tssh` alias - Cluster SSH replacement + The `tmux::tssh_from_argument` helper + The `tmux::tssh_from_file` helper + `tssh` examples + Common Tmux commands I use in `tssh` + Copy and paste workflow + Tmux configurations +``` + +## Introduction + +Tmux (Terminal Multiplexer) is a powerful, terminal-based tool that manages multiple terminal sessions within a single window. Here are some of its primary features and functionalities: + +* Session management +* Window and Pane management +* Persistent Workspace +* Customization + +[https://github.com/tmux/tmux/wiki](https://github.com/tmux/tmux/wiki) + +Before continuing to read this post, I encourage you to get familiar with Tmux first (unless you already know the basics). You can go through the official getting started guide: + +[https://github.com/tmux/tmux/wiki/Getting-Started](https://github.com/tmux/tmux/wiki/Getting-Started) + +I can also recommend this book (this is the book I got started with with Tmux): + +[https://pragprog.com/titles/bhtmux2/tmux-2/](https://pragprog.com/titles/bhtmux2/tmux-2/) + +Over the years, I have built a couple of shell helper functions to optimize my workflows. Tmux is extensively integrated into my daily workflows (personal and work). I had colleagues asking me about my Tmux config and helper scripts for Tmux several times. It would be neat to blog about it so that everyone interested in it can make a copy of my configuration and scripts. + +The configuration and scripts in this blog post are only the non-work-specific parts. There are more helper scripts, which I only use for work (and aren't really useful outside of work due to the way servers and clusters are structured there). + +Tmux is highly configurable, and I think I am only scratching the surface of what is possible with it. Nevertheless, it may still be useful for you. I also love that Tmux is part of the OpenBSD base system! + +## Shell aliases + +I am a user of the Z-Shell (`zsh`), but I believe all the snippets mentioned in this blog post also work with Bash. + +[https://www.zsh.org](https://www.zsh.org) + +For the most common Tmux commands I use, I have created the following shell aliases: + +```bash +alias tm=tmux +alias tl='tmux list-sessions' +alias tn=tmux::new +alias ta=tmux::attach +alias tx=tmux::remote +alias ts=tmux::search +alias tssh=tmux::cluster_ssh +``` + +Note all `tmux::...`; those are custom shell functions doing certain things, and they aren't part of the Tmux distribution. But let's run through every aliases one by one. + +The first two are pretty straightforward. `tm` is simply a shorthand for `tmux`, so I have to type less, and `tl` lists all Tmux sessions that are currently open. No magic here. + +## The `tn` alias - Creating a new session + +The `tn` alias is referencing this function: + +```bash +# Create new session and if alread exists attach to it +tmux::new () { + readonly session=$1 + local date=date + if where gdate &>/dev/null; then + date=gdate + fi + + tmux::cleanup_default + if [ -z "$session" ]; then + tmux::new T$($date +%s) + else + tmux new-session -d -s $session + tmux -2 attach-session -t $session || tmux -2 switch-client -t $session + fi +} +alias tn=tmux::new +``` + +There is a lot going on here. Let's have a detailed look at what it is doing. As a note, the function relies on GNU Date, so MacOS is looking for the `gdate` commands to be available. Otherwise, it will fall back to `date`. You need to install GNU Date for Mac, as it isn't installed by default there. As I use Fedora Linux on my personal Laptop and a MacBook for work, I have to make it work for both. + +First, a Tmux session name can be passed to the function as a first argument. That session name is only optional. Without it, Tmux will select a session named `T$($date +%s)` as a default. Which is T followed by the UNIX epoch, e.g. `T1717133796`. + +### Cleaning up default sessions automatically + +Note also the call to `tmux::cleanup_default`; it would clean up all already opened default sessions if they aren't attached. Those sessions were only temporary, and I had too many flying around after a while. So, I decided to auto-delete the sessions if they weren't attached. If I want to keep sessions around, I will rename them with the Tmux command `prefix-key $`. This is the cleanup function: + +```bash +tmux::cleanup_default () { + local s + tmux list-sessions | grep '^T.*: ' | grep -F -v attached | + cut -d: -f1 | while read -r s; do + echo "Killing $s" + tmux kill-session -t "$s" + done +} +``` + +The cleanup function kills all open Tmux sessions that haven't been renamed properly yet—but only if they aren't attached (e.g., don't run in the foreground in any terminal). Cleaning them up automatically keeps my Tmux sessions as neat and tidy as possible. + +### Renaming sessions + +Whenever I am in a temporary session (named `T....`), I may decide that I want to keep this session around. I have to rename the session to prevent the cleanup function from doing its thing. That's, as mentioned already, easily accomplished with the standard `prefix-key $` Tmux command. + +## The `ta` alias - Attaching to a session + +This alias refers to the following function, which tries to attach to an already-running Tmux session. + +```bash +tmux::attach () { + readonly session=$1 + + if [ -z "$session" ]; then + tmux attach-session || tmux::new + else + tmux attach-session -t $session || tmux::new $session + fi +} +alias ta=tmux::attach +``` + +If no session is specified (as the argument of the function), it will try to attach to the first open session. If no Tmux server is running, it will create a new one with `tmux::new`. Otherwise, with a session name given as the argument, it will attach to it. If unsuccessful (e.g., the session doesn't exist), it will be created and attached to. + +## The `tr` alias - For a nested remote session + +This SSHs into the remote server specified and then, remotely on the server itself, starts a nested Tmux session. So we have one Tmux session on the local computer and, inside of it, an SSH connection to a remote server with a Tmux session running again. The benefit of this is that, in case my network connection breaks down, the next time I connect, I can continue my work on the remote server exactly where I left off. The session name is the name of the server being SSHed into. If a session like this already exists, it simply attaches to it. + +```bash +tmux::remote () { + readonly server=$1 + tmux new -s $server "ssh -t $server 'tmux attach-session || tmux'" || \ + tmux attach-session -d -t $server +} +alias tr=tmux::remote +``` + +### Change of the Tmux prefix for better nesting + +To make nested Tmux sessions work smoothly, one must change the Tmux prefix key locally or remotely. By default, the Tmux prefix key is `Ctrl-b`, so `Ctrl-b $`, for example, renames the current session. To change the prefix key from the standard `Ctrl-b` to, for example, `Ctrl-g`, you must add this to the `tmux.conf`: + +``` +set-option -g prefix C-g +``` + +This way, when I want to rename the remote Tmux session, I have to use `Ctrl-g $`, and when I want to rename the local Tmux session, I still have to use `Ctrl-b $`. In my case, I have this deployed to all remote servers through a configuration management system (out of scope for this blog post). + +There might also be another way around this (without reconfiguring the prefix key), but that is cumbersome to use, as far as I remember. + +## The `ts` alias - Searching sessions with fuzzy finder + +Despite the fact that with `tmux::cleanup_default`, I don't leave a huge mess with trillions of Tmux sessions flying around all the time, at times, it can become challenging to find exactly the session I am currently interested in. After a busy workday, I often end up with around twenty sessions on my laptop. This is where fuzzy searching for session names comes in handy, as I often don't remember the exact session names. + +```bash +tmux::search () { + local -r session=$(tmux list-sessions | fzf | cut -d: -f1) + if [ -z "$TMUX" ]; then + tmux attach-session -t $session + else + tmux switch -t $session + fi +} +alias ts=tmux::search +``` + +All it does is list all currently open sessions in `fzf`, where one of them can be searched and selected through fuzzy find, and then either switch (if already inside a session) to the other session or attach to the other session (if not yet in Tmux). + +You must install the `fzf` command on your computer for this to work. This is how it looks like: + +[](./terminal-multiplexing-with-tmux/tmux-session-fzf.png) + +## The `tssh` alias - Cluster SSH replacement + +Before I used Tmux, I was a heavy user of ClusterSSH, which allowed me to log in to multiple servers at once in a single terminal window and type and run commands on all of them in parallel. + +[https://github.com/duncs/clusterssh](https://github.com/duncs/clusterssh) + +However, since I started using Tmux, I retired ClusterSSH, as it came with the benefit that Tmux only needs to be run in the terminal, whereas ClusterSSH spawned terminal windows, which aren't easily portable (e.g., from a Linux desktop to macOS). The `tmux::cluster_ssh` function can have N arguments, where: + +* ...the first argument will be the session name (see `tmux::tssh_from_argument` helper function), and all remaining arguments will be server hostnames/FQDNs to connect to simultaneously. +* ...or, the first argument is a file name, and the file contains a list of hostnames/FQDNs (see `tmux::ssh_from_file` helper function) + +This is the function definition behind the `tssh` alias: + +```bash +tmux::cluster_ssh () { + if [ -f "$1" ]; then + tmux::tssh_from_file $1 + return + fi + + tmux::tssh_from_argument $@ +} +alias tssh=tmux::cluster_ssh +``` + +This function is just a wrapper around the more complex `tmux::tssh_from_file` and `tmux::tssh_from_argument` functions, as you have learned already. Most of the magic happens there. + +### The `tmux::tssh_from_argument` helper + +This is the most magic helper function we will cover in this post. It looks like this: + +```bash +tmux::tssh_from_argument () { + local -r session=$1; shift + local first_server=$1; shift + + tmux new-session -d -s $session "ssh -t $first_server" + if ! tmux list-session | grep "^$session:"; then + echo "Could not create session $session" + return 2 + fi + + for server in "${@[@]}"; do + tmux split-window -t $session "tmux select-layout tiled; ssh -t $server" + done + + tmux setw -t $session synchronize-panes on + tmux -2 attach-session -t $session | tmux -2 switch-client -t $session +} +``` + +It expects at least two arguments. The first argument is the session name to create for the clustered SSH session. All other arguments are server hostnames or FQDNs to which to connect. The first one is used to make the initial session. All remaining ones are added to that session with `tmux split-window -t $session...`. At the end, we enable synchronized panes by default, so whenever you type, the commands will be sent to every SSH connection, thus allowing the neat ClusterSSH feature to run commands on multiple servers simultaneously. Once done, we attach (or switch, if already in Tmux) to it. + +Sometimes, I don't want the synchronized panes behavior and want to switch it off temporarily. I can do that with `prefix-key p` and `prefix-key P` after adding the following to my local `tmux.conf`: + +``` +bind-key p setw synchronize-panes off +bind-key P setw synchronize-panes on +``` + +### The `tmux::tssh_from_file` helper + +This one sets the session name to the file name and then reads a list of servers from that file, passing the list of servers to `tmux::tssh_from_argument` as the arguments. So, this is a neat little wrapper that also enables me to open clustered SSH sessions from an input file. + +```bash +tmux::tssh_from_file () { + local -r serverlist=$1; shift + local -r session=$(basename $serverlist | cut -d. -f1) + + tmux::tssh_from_argument $session $(awk '{ print $1} ' $serverlist | sed 's/.lan./.lan/g') +} +``` + +### `tssh` examples + +To open a new session named `fish` and log in to 4 remote hosts, run this command (Note that it is also possible to specify the remote user): + +``` +$ tssh fish blowfish.buetow.org fishfinger.buetow.org \ + fishbone.buetow.org user@octopus.buetow.org +``` + +To open a new session named `manyservers`, put many servers (one FQDN per line) into a file called `manyservers.txt` and simply run: + +``` +$ tssh manyservers.txt +``` + +### Common Tmux commands I use in `tssh` + +These are default Tmux commands that I make heavy use of in a `tssh` session: + +* Press `prefix-key DIRECTION` to switch panes. DIRECTION is by default any of the arrow keys, but I also configured Vi keybindings. +* Press `prefix-key <space>` to change the pane layout (can be pressed multiple times to cycle through them). +* Press `prefix-key z` to zoom in and out of the current active pane. + +## Copy and paste workflow + +As you will see later in this blog post, I have configured a history limit of 1 million items in Tmux so that I can scroll back quite far. One main workflow of mine is to search for text in the Tmux history, select and copy it, and then switch to another window or session and paste it there (e.g., into my text editor to do something with it). + +This works by pressing `prefix-key [` to enter Tmux copy mode. From there, I can browse the Tmux history of the current window using either the arrow keys or vi-like navigation (see vi configuration later in this blog post) and the Pg-Dn and Pg-Up keys. + +I often search the history backwards with `prefix-key [` followed by a `?`, which opens the Tmux history search prompt. + +Once I have identified the terminal text to be copied, I enter visual select mode with `v`, highlight all the text to be copied (using arrow keys or Vi motions), and press `y` to yank it (sorry if this all sounds a bit complicated, but Vim/NeoVim users will know this, as it is pretty much how you do it there as well). + +For `v` and `y` to work, the following has to be added to the Tmux configuration file: + +``` +bind-key -T copy-mode-vi 'v' send -X begin-selection +bind-key -T copy-mode-vi 'y' send -X copy-selection-and-cancel +``` + +Once the text is yanked, I switch to another Tmux window or session where, for example, a text editor is running and paste the yanked text from Tmux into the editor with `prefix-key ]`. Note that when pasting into a modal text editor like Vi or Helix, you would first need to enter insert mode before `prefix-key ]` would paste anything. + +## Tmux configurations + +Some features I have configured directly in Tmux don't require an external shell alias to function correctly. Let's walk line by line through my local `~/config/tmux/tmux.conf`: + +``` +source ~/.config/tmux/tmux.local.conf + +set-option -g allow-rename off +set-option -g history-limit 100000 +set-option -g status-bg '#444444' +set-option -g status-fg '#ffa500' +set-option -s escape-time 0 +``` + +There's yet to be much magic happening here. I source a `tmux.local.conf`, which I sometimes use to override the default configuration that comes from the configuration management system. But it is mostly just an empty file, so it doesn't throw any errors on Tmux startup when I don't use it. + +I work with many terminal outputs, which I also like to search within Tmux. So, I added a large enough `history-limit`, enabling me to search backwards in Tmux for any output up to a million lines of text. + +Besides changing some colours (personal taste), I also set `escape-time` to `0`, which is just a workaround. Otherwise, my Helix text editor's `ESC` key would take ages to trigger within Tmux. I am trying to remember the gory details. You can leave it out; if everything works fine for you, leave it out. + +The next lines in the configuration file are: + +``` +set-window-option -g mode-keys vi +bind-key -T copy-mode-vi 'v' send -X begin-selection +bind-key -T copy-mode-vi 'y' send -X copy-selection-and-cancel +``` + +I navigate within Tmux using Vi keybindings, so the `mode-keys` is set to `vi`. I use the Helix modal text editor, which is close enough to Vi bindings for simple navigation to feel "native" to me. (By the way, I have been a long-time Vim and NeoVim user, but I eventually switched to Helix. It's off-topic here, but it may be worth another blog post once.) + +The two `bind-key` commands make it so that I can use `v` and `y` in copy mode, which feels more Vi-like (as already discussed earlier in this post). + +The next set of lines in the configuration file are: + +``` +bind-key h select-pane -L +bind-key j select-pane -D +bind-key k select-pane -U +bind-key l select-pane -R + +bind-key H resize-pane -L 5 +bind-key J resize-pane -D 5 +bind-key K resize-pane -U 5 +bind-key L resize-pane -R 5 +``` + +These allow me to use `prefix-key h`, `prefix-key j`, `prefix-key k`, and `prefix-key l` for switching panes and `prefix-key H`, `prefix-key J`, `prefix-key K`, and `prefix-key L` for resizing the panes. If you don't know Vi/Vim/NeoVim, the letters `hjkl` are commonly used there for left, down, up, and right, which is also the same for Helix, by the way. + +The next set of lines in the configuration file are: + +``` +bind-key c new-window -c '#{pane_current_path}' +bind-key F new-window -n "session-switcher" "tmux list-sessions | fzf | cut -d: -f1 | xargs tmux switch-client -t" +bind-key T choose-tree +``` + +The first one is that any new window starts in the current directory. The second one is more interesting. I list all open sessions in the fuzzy finder. I rely heavily on this during my daily workflow to switch between various sessions depending on the task. E.g. from a remote cluster SSH session to a local code editor. + +The third one, `choose-tree`, opens a tree view in Tmux listing all sessions and windows. This one is handy to get a better overview of what is currently running in any local Tmux session. It looks like this (it also allows me to press a hotkey to switch to a particular Tmux window): + +[](./terminal-multiplexing-with-tmux/tmux-tree-view.png) + + +The last remaining lines in my configuration file are: + +``` +bind-key p setw synchronize-panes off +bind-key P setw synchronize-panes on +bind-key r source-file ~/.config/tmux/tmux.conf \; display-message "tmux.conf reloaded" +``` + +We discussed `synchronized panes` earlier. I use it all the time in clustered SSH sessions. When enabled, all panes (remote SSH sessions) receive the same keystrokes. This is very useful when you want to run the same commands on many servers at once, such as navigating to a common directory, restarting a couple of services at once, or running tools like `htop` to quickly monitor system resources. + +The last one reloads my Tmux configuration on the fly. + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +[Back to the main site](../) diff --git a/gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.md b/gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.md deleted file mode 100644 index 3e388ffe..00000000 --- a/gemfeed/DRAFT-KISS-high-availability-with-OpenBSD.md +++ /dev/null @@ -1,286 +0,0 @@ -# KISS high-availability with OpenBSD - -``` -Art by Michael J. Penick (mod. by Paul B) - - __________ - / nsd tower\ ( - /____________\ (\) awk-ward - |:_:_:_:_:_| )) plant - |_:_,--.:_:| dig-bubble (\// ) - |:_:|__|_:_| relayd-castle _ ) )) (( - _ |_ _ :_:| _ _ _ (_) (((( /)\` - | |_| |_| | _| | |_| |_| | o \\)) (( ( - \_:_:_:_:/|_|_|_|\:_:_:_:_/ . (( )))) - |_,-._:_:_:_:_:_:_:_.-,_| )) ((// - |:|_|:_:_:,---,:_:_:|_|:| ,-. )/ - |_:_:_:_,'puffy `,_:_:_:_| _ o ,;'))(( - |:_:_:_/ _ | _ \_:_:_:| (_O (( )) -_____|_:_:_| (o)-(o) |_:_:_|--'`-. ,--. ksh under-water (((\'/ - ', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\ /,---.~ goat \`)) -. ` |_:_:_| \`-'/ |_:_:_|. ` . ` /()\.__( ) .,-----'`-\(( sed-root - ', ;|:_:_:| `-' |:_:_:| ', ; ', ; `--'| \ ', ; ', ; ',')).,-- -. ` MJP ` . ` . ` . ` . httpd-soil ` . . ` . ` . ` . ` . ` - ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; - -``` - -I have always wanted a highly available setup for my personal websites. I could have used off-the-shelf hosting solutions or hosted my sites in an AWS S3 bucket. I have used technologies like BGP, LVS/IPVS, ldirectord, Pacemaker, heartbeat, heartbeat2, Corosync, keepalived, DRBD, and commercial F5 Load Balancers for high availability at work. - -But still, my personal sites were never highly available. All those technologies are great for professional use, but I was looking for something much more straightforward for my personal space—something as KISS (keep it simple and stupid) as possible. - -It would be fine if my personal website wasn't highly available, but the geek in me wants it anyway. - -> PS: ASCII-art reflects the OpenBSD under-water world with all the tools available in the base system. - -## My auto-failover requirements - -* Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible. -* Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month). -* It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers. -* It should be geo-redundant. -* It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server. -* Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available. -* Let's Encrypt TLS certificates should always work (before and after a failover). -* Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover. -* Don't configure everything manually. The configuration should be automated and reproducible. - -## My HA solution - -### Only OpenBSD base installation required - -My HA solution for Web and Gemini is based on DNS (OpenBSD's `nsd`) and a simple shell script (OpenBSD's `ksh` and some little `sed` and `awk` and `grep`). All software used here is part of the OpenBSD base system and no external package needs to be installed - OpenBSD is a complete operating system. - -[https://man.OpenBSD.org/nsd.8](https://man.OpenBSD.org/nsd.8) -[https://man.OpenBSD.org/ksh](https://man.OpenBSD.org/ksh) -[https://man.OpenBSD.org/awk](https://man.OpenBSD.org/awk) -[https://man.OpenBSD.org/sed](https://man.OpenBSD.org/sed) -[https://man.OpenBSD.org/dig](https://man.OpenBSD.org/dig) -[https://man.OpenBSD.org/ftp](https://man.OpenBSD.org/ftp) - -I also used the `dig` (for DNS checks) and `ftp` (for HTTP/HTTPS checks) programs. - -The DNS failover is performed automatically between the two OpenBSD VMs involved (my setup doesn't require any quorum for a failover, so there isn't a need for a 3rd VM). The `ksh` script, executed once per minute via CRON (on both VMs), performs a health check to determine whether the current master node is available. If the current master isn't available (no HTTP response as expected), a failover is performed to the standby VM: - -```sh -#!/bin/ksh - -ZONES_DIR=/var/nsd/zones/master/ -DEFAULT_MASTER=fishfinger.buetow.org -DEFAULT_STANDBY=blowfish.buetow.org - -determine_master_and_standby () { - local master=$DEFAULT_MASTER - local standby=$DEFAULT_STANDBY - - . - . - . - - local -i health_ok=1 - if ! ftp -4 -o - https://$master/index.txt | grep -q "Welcome to $master"; then - echo "https://$master/index.txt IPv4 health check failed" - health_ok=0 - elif ! ftp -6 -o - https://$master/index.txt | grep -q "Welcome to $master"; then - echo "https://$master/index.txt IPv6 health check failed" - health_ok=0 - fi - if [ $health_ok -eq 0 ]; then - local tmp=$master - master=$standby - standby=$tmp - fi - - . - . - . -} -``` - -The failover scripts looks for the ` ; Enable failover` string in the DNS zone files and swaps the `A` and `AAAA` records of the DNS entries accordingly: - -```sh -fishfinger$ grep failover /var/nsd/zones/master/foo.zone.zone - 300 IN A 46.23.94.99 ; Enable failover - 300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover -www 300 IN A 46.23.94.99 ; Enable failover -www 300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover -standby 300 IN A 23.88.35.144 ; Enable failover -standby 300 IN AAAA 2a01:4f8:c17:20f1::42 ; Enable failover -``` - -```sh -tramsform () { - sed -E ' - /IN A .*; Enable failover/ { - /^standby/! { - s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/master_a)' ; \3/; - } - /^standby/ { - s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/standby_a)' ; \3/; - } - } - /IN AAAA .*; Enable failover/ { - /^standby/! { - s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/master_aaaa)' ; \3/; - } - /^standby/ { - s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/standby_aaaa)' ; \3/; - } - } - / ; serial/ { - s/^( +) ([0-9]+) .*; (.*)/\1 '$(date +%s)' ; \3/; - } - ' -} -``` - -After the failover, the script reloads `nsd` and performs a sanity check to see if DNS still works. If not, a rollback will be performed: - -```sh -# Race condition (e.g. script execution abored in the middle of the previous run) -if [ -f $zone_file.bak ]; then - mv $zone_file.bak $zone_file -fi - -cat $zone_file | transform > $zone_file.new.tmp - -grep -v ' ; serial' $zone_file.new.tmp > $zone_file.new.noserial.tmp -grep -v ' ; serial' $zone_file > $zone_file.old.noserial.tmp - -echo "Has zone $zone_file changed?" -if diff -u $zone_file.old.noserial.tmp $zone_file.new.noserial.tmp; then - echo "The zone $zone_file hasn't changed" - rm $zone_file.*.tmp - return 0 -fi - -cp $zone_file $zone_file.bak -mv $zone_file.new.tmp $zone_file -rm $zone_file.*.tmp -echo "Reloading nsd" -nsd-control reload - -if ! zone_is_ok $zone; then - echo "Rolling back $zone_file changes" - cp $zone_file $zone_file.invalid - mv $zone_file.bak $zone_file - echo "Reloading nsd" - nsd-control reload - zone_is_ok $zone - return 3 -fi - -for cleanup in invalid bak; do - if [ -f $zone_file.$cleanup ]; then - rm $zone_file.$cleanup - fi -done - -echo "Failover of zone $zone to $MASTER completed" -return 1 -``` - -A non-zero return code (here, 3 when a rollback and 1 when a DNS failover was performed) will cause CRON to send an E-Mail with the whole script output. - -The nameserver is running on both VMs, and both are configured to be "master" DNS servers so that they have their own individual zone files, which can be changed independently. Otherwise, my setup wouldn't work. The side effect is that under a split-brain scenario (both VMs cannot see each other), both would promote themselves to master via their local DNS entries. More about that later, but that's fine in my use case. - -Check out the whole script here: - -[https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh](https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/dns-failover.ksh) - -### Fairly cheap and geo-redundant - -I am renting two small OpenBSD VMs: One at OpenBSD Amsterdam and the other at Hetzner Cloud. So, both VMs are hosted at another provider, in different IP subnets, and in different countries (the Netherlands and Germany). - -[https://openbsd.amsterdam](https://openbsd.amsterdam) -[https://www.hetzner.cloud](https://www.hetzner.cloud) - -I only have a little traffic on my sites. I could always upload the static content to AWS S3 if I suddenly had to. But this will never be required. - -A DNS-based failover is cheap, as there isn't any BGP or fancy load balancer to pay for. Small VMs also cost less than millions. - -### Failover time and split-brain - -A DNS failover doesn't happen immediately. I've configured a DNS TTL of `300` seconds, and the failover script checks once per minute whether to perform a failover or not. So, in total, a failover can take six minutes (not including other DNS caching servers somewhere in the interweb, but that's fine - eventually, all requests will resolve to the new master after a failover). - -A split-brain scenario between the old master and the new master might happen. That's OK, as my sites are static, and there's no database to synchronise other than HTML, CSS, and images when the site is updated. - -### Failover support for multiple protocols - -With the DNS failover, HTTP, HTTPS, and Gemini protocols are failovered. This works because all domain virtual hosts are configured on either VM's `httpd` (OpenBSD's HTTP server) and `relayd` (it's also part of OpenBSD and I use it to TLS offload the Gemini protocol). So, both VMs accept requests for all the hosts. It's just a matter of the DNS entry, which hosts receive the requests. - -[https://man.openbsd.org/httpd.8](https://man.openbsd.org/httpd.8) -[https://man.openbsd.org/relayd.8](https://man.openbsd.org/relayd.8) - -For example, the master is responsible for the `https://www.foo.zone` and `https://foo.zone` hosts, whereas the standby can be reached via `https://standby.foo.zone` (port 80 for plain HTTP works as well). The same principle is followed with all the other hosts, e.g. `irregular.ninja`, `paul.buetow.org` and so on. The same applies to my Gemini capsules for `gemini://foo.zone`, `gemini://standby.foo.zone`, `gemini://paul.buetow.org` and `gemini://standby.paul.buetow.org`. - -On DNS failover, master and standby swap roles without config changes other than the DNS entries. That's KISS (keep it simple and stupid)! - -### Let's encrypt TLS certificates - -All my hosts use TLS certificates from Let's Encrypt. The ACME automation for requesting and keeping the certificates valid (up to date) requires that the host requesting a certificate from Let's Encrypt is also the host using that certificate. - -If the master always serves `foo.zone` and the standby always `standby.foo.zone`, then there would be a problem after the failover, as the new master wouldn't have a valid certificate for `foo.zone` and the new standby wouldn't have a valid certificate for `standby.foo.zone` which would lead to TLS errors on the clients. - -As a solution, the CRON job responsible for the DNS failover also checks for the current week number of the year so that: - -* In an odd week number, the first server is the default master -* In an even week number, the second server is the default master. - -Which translates to: - -```sh -# Weekly auto-failover for Let's Encrypt automation -local -i -r week_of_the_year=$(date +%U) -if [ $(( week_of_the_year % 2 )) -eq 0 ]; then - local tmp=$master - master=$standby - standby=$tmp -fi -``` - -This way, a DNS failover is performed weekly so that the ACME automation can update the Let's Encrypt certificates (for master and standby) before they expire on each VM. - -The ACME automation is yet another daily CRON script `/usr/local/bin/acme.sh`. It iterates over all of my Let's Encrypt hosts, checks whether they resolve to the same IP address as the current VM, and only then invokes the ACME client to request or renew the TLS certificates. So, there are always correct requests made to Let's Encrypt. - -Let's encrypt certificates usually expire after 3 months, so a weekly failover of my VMs is plenty. - -[`acme.sh.tpl` - Rex template for the `acme.sh` script of mine.](https://codeberg.org/snonux/rexfiles/src/branch/master/frontends/scripts/acme.sh.tpl) -[https://man.openbsd.org/acme-client.1](https://man.openbsd.org/acme-client.1) -[Let's Encrypt with OpenBSD and Rex](./2022-07-30-lets-encrypt-with-openbsd-and-rex.md) - -### Monitoring - -CRON is sending me an E-Mail whenever a failover is performed (or whenever a failover failed). Furthermore, I am monitoring my DNS servers and hosts through Gogios, the monitoring system I have developed. - -[https://codeberg.org/snonux/gogios](https://codeberg.org/snonux/gogios) -[KISS server monitoring with Gogios](./2023-06-01-kiss-server-monitoring-with-gogios.md) - -### Rex automation - -I use Rexify, a friendly configuration management system that allows automatic deployment and configuration. - -[https://www.rexify.org](https://www.rexify.org) -[https://codeberg.org/snonux/rexfiles/src/branch/master/frontends](https://codeberg.org/snonux/rexfiles/src/branch/master/frontends) - -## More HA - -Other high-available services running on my OpenBSD VMs are my MTAs for mail forwarding (OpenSMTPD) and the authoritative DNS servers (`nsd`) for all my domains. No particular HA setup is required, though, as the protocols (SMTP and DNS) already take care of the failover to the next available host! - -As a password manager, I use `geheim`, a command-line tool I wrote in Ruby with encrypted files in a git repository (I even have it installed in Termux on my Phone). For HA reasons, I simply updated the client code so that it always synchronises the database with both servers when I run the `sync` command there. - -[https://codeberg.org/snonux/geheim](https://codeberg.org/snonux/geheim) - -E-Mail your comments to `paul@nospam.buetow.org` :-) - -Other *BSD and KISS related posts are: - -[2016-04-09 Jails and ZFS with Puppet on FreeBSD](./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md) -[2022-07-30 Let's Encrypt with OpenBSD and Rex](./2022-07-30-lets-encrypt-with-openbsd-and-rex.md) -[2022-10-30 Installing DTail on OpenBSD](./2022-10-30-installing-dtail-on-openbsd.md) -[2023-06-01 KISS server monitoring with Gogios](./2023-06-01-kiss-server-monitoring-with-gogios.md) -[2023-10-29 KISS static web photo albums with `photoalbum.sh`](./2023-10-29-kiss-static-web-photo-albums-with-photoalbum.sh.md) -[2024-01-13 One reason why I love OpenBSD](./2024-01-13-one-reason-why-i-love-openbsd.md) - -[Back to the main site](../) diff --git a/gemfeed/index.md b/gemfeed/index.md index ba0c2b4a..d192bfbd 100644 --- a/gemfeed/index.md +++ b/gemfeed/index.md @@ -2,6 +2,7 @@ ## To be in the .zone! +[2024-06-23 - Terminal multiplexing with `tmux`](./2024-06-23-terminal-multiplexing-with-tmux.md) [2024-05-03 - Projects I currently don't have time for](./2024-05-03-projects-i-currently-dont-have-time-for.md) [2024-05-01 - 'Slow Productivity' book notes](./2024-05-01-slow-productivity-book-notes.md) [2024-04-01 - KISS high-availability with OpenBSD](./2024-04-01-KISS-high-availability-with-OpenBSD.md) diff --git a/gemfeed/terminal-multiplexing-with-tmux/tmux-session-fzf.png b/gemfeed/terminal-multiplexing-with-tmux/tmux-session-fzf.png Binary files differnew file mode 100644 index 00000000..7a2e9440 --- /dev/null +++ b/gemfeed/terminal-multiplexing-with-tmux/tmux-session-fzf.png diff --git a/gemfeed/terminal-multiplexing-with-tmux/tmux-tree-view.png b/gemfeed/terminal-multiplexing-with-tmux/tmux-tree-view.png Binary files differnew file mode 100644 index 00000000..672859c5 --- /dev/null +++ b/gemfeed/terminal-multiplexing-with-tmux/tmux-tree-view.png @@ -1,6 +1,6 @@ # foo.zone -> This site was generated at 2024-05-18T13:34:09+03:00 by `Gemtexter` +> This site was generated at 2024-06-23T23:06:46+03:00 by `Gemtexter` ``` |\---/| @@ -32,6 +32,7 @@ If you reach this site via the modern web, please read this: ### Posts +[2024-06-23 - Terminal multiplexing with `tmux`](./gemfeed/2024-06-23-terminal-multiplexing-with-tmux.md) [2024-05-03 - Projects I currently don't have time for](./gemfeed/2024-05-03-projects-i-currently-dont-have-time-for.md) [2024-05-01 - 'Slow Productivity' book notes](./gemfeed/2024-05-01-slow-productivity-book-notes.md) [2024-04-01 - KISS high-availability with OpenBSD](./gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md) diff --git a/uptime-stats.md b/uptime-stats.md index 52f10860..46031859 100644 --- a/uptime-stats.md +++ b/uptime-stats.md @@ -1,6 +1,6 @@ # My machine uptime stats -> This site was last updated at 2024-05-18T13:34:09+03:00 +> This site was last updated at 2024-06-23T23:06:46+03:00 The following stats were collected via `uptimed` on all of my personal computers over many years and the output was generated by `guprecords`, the global uptime records stats analyser of mine. |
