foo.zone feed

Random Weird Things - Part Ⅱ

2025-02-08T11:06:16+02:00

Random Weird Things - Part Ⅱ

+
+Published at 2025-02-08T11:06:16+02:00
+
+Every so often, I come across random, weird, and unexpected things on the internet. I thought it would be neat to share them here from time to time. This is the second run.
+
+2024-07-05 Random Weird Things - Part Ⅰ
+2025-02-08 Random Weird Things - Part Ⅱ (You are currently reading this)
+
+

+/\_/\           /\_/\
+( o.o ) WHOA!! ( o.o )
+> ^ <           > ^ <
+/   \    MOEEW! /   \
+/______\       /______\
+

+
+

Random Weird Things - Part Ⅱ
⇢ 11. The SQLite codebase is a gem
⇢ Go Programming
⇢ ⇢ 12. Official Go font
⇢ ⇢ 13. Go functions can have methods
⇢ macOS
⇢ ⇢ 14. ß and ss are treated the same
⇢ ⇢ 15. Colon as file path separator
⇢ 16. Polyglots - programs written in multiple languages
⇢ 17. Languages, where indices start at 1
⇢ 18. Perl Poetry
⇢ 19. CSS3 is turing complete
⇢ 20. The biggest shell programs

11. The SQLite codebase is a gem

+
+Check this out:
+
+
+
+Source:
+
+https://wetdry.world/@memes/112717700557038278
+
+

Go Programming

+
+

12. Official Go font

+
+The Go programming language has an official font called "Go Font." It was created to complement the aesthetic of the Go language, ensuring clear and legible rendering of code. The font includes a monospace version for code and a proportional version for general text, supporting consistent look and readability in Go-related materials and development environments.
+
+Check out some Go code displayed using the Go font:
+
+
+
+https://go.dev/blog/go-fonts
+
+The design emphasizes simplicity and readability, reflecting Go's philosophy of clarity and efficiency.
+
+I found it interesting and/or weird, as Go is a programming language. Why should it bother having its own font? I have never seen another open-source project like Go do this. But I also like it. Maybe I will use it in the future for this blog :-)
+
+

13. Go functions can have methods

+
+Functions on struct types? Well, know. Functions on types like int and string? It's also known of, but a bit lesser. Functions on function types? That sounds a bit funky, but it's possible, too! For demonstration, have a look at this snippet:
+
+ +

package main
+
+import "log"
+
+type fun func() string
+
+func (f fun) Bar() string {
+        return "Bar"
+}
+
+func main() {
+        var f fun = func() string {
+                return "Foo"
+        }
+        log.Println("Example 1: ", f())
+        log.Println("Example 2: ", f.Bar())
+        log.Println("Example 3: ", fun(f.Bar).Bar())
+        log.Println("Example 4: ", fun(fun(f.Bar).Bar).Bar())
+}
+

+
+It runs just fine:
+
+ +

❯ go run main.go
+2025/02/07 22:56:14 Example 1:  Foo
+2025/02/07 22:56:14 Example 2:  Bar
+2025/02/07 22:56:14 Example 3:  Bar
+2025/02/07 22:56:14 Example 4:  Bar
+

+
+

macOS

+
+For personal computing, I don't use Apple, but I have to use it for work.
+
+

14. ß and ss are treated the same

+
+Know German? In German, the letter "sarp s" is written as ß. ß is treated the same as ss on macOS.
+
+On a case-insensitive file system like macOS, not only are uppercase and lowercase letters treated the same, but non-Latin characters like the German "ß" are also considered equivalent to their Latin counterparts (in this case, "ss").
+
+So, even though "Maß" and "Mass" are not strictly equivalent, the macOS file system still treats them as the same filename due to its handling of Unicode characters. This can sometimes lead to unexpected behaviour. Check this out:
+
+ +

❯ touch Maß
+❯ ls -l
+-rw-r--r--@ 1 paul  wheel  0 Feb  7 23:02 Maß
+❯ touch Mass
+❯ ls -l
+-rw-r--r--@ 1 paul  wheel  0 Feb  7 23:02 Maß
+❯ rm Mass
+❯ ls -l
+
+❯ touch Mass
+❯ ls -ltr
+-rw-r--r--@ 1 paul  wheel  0 Feb  7 23:02 Mass
+❯ rm Maß
+❯ ls -l
+
+

+
+

15. Colon as file path separator

+
+MacOS can use the colon as a file path separator on its ADFS (file system). A typical ADFS file pathname on a hard disc might be:
+
+

+ADFS::4.$.Documents.Techwriter.Myfile
+

+
+I can't reproduce this on my (work) Mac, though, as it now uses the APFS file system. In essence, ADFS is an older file system, while APFS is a contemporary file system optimized for Apple's modern devices.
+
+https://social.jvns.ca/@b0rk/113041293527832730
+
+

16. Polyglots - programs written in multiple languages

+
+A coding polyglot is a program or script written so that it can be executed in multiple programming languages without modification. This is typically achieved by leveraging syntax overlaps or crafting valid and meaningful code in each targeted language. Polyglot programs are often created as a challenge or for demonstration purposes to showcase language similarities or clever coding techniques.
+
+Check out my very own polyglot:
+
+The fibonatti.pl.c Polyglot
+
+

17. Languages, where indices start at 1

+
+Array indices start at 1 instead of 0 in some programming languages, known as one-based indexing. This can be controversial because zero-based indexing is more common in popular languages like C, C++, Java, and Python. One-based indexing can lead to off-by-one errors when developers switch between languages with different indexing schemes.
+
+Languages with One-Based Indexing:
+
+

Fortran
MATLAB
Lua
R (for vectors and lists)
Smalltalk
Julia (by default, although zero-based indexing is also possible)

+foo.lua example:
+
+ +

arr = {10, 20, 30, 40, 50}
+print(arr[1]) -- Accessing the first element
+

+
+ +

❯ lua foo.lua
+10
+

+
+One-based indexing is more natural for human-readable, mathematical, and theoretical contexts, where counting traditionally starts from one.
+
+

18. Perl Poetry

+
+Perl Poetry is a playful and creative practice within the programming community where Perl code is written as a poem. These poems are crafted to be syntactically valid Perl code and make sense as poetic text, often with whimsical or humorous intent. This showcases Perl's flexibility and expressiveness, as well as the creativity of its programmers.
+
+See this Poetry of my own; the Perl interpreter does not yield any syntax error parsing that. But also, the Peom doesn't do anything useful then executed:
+
+ +

# (C) 2006 by Paul C. Buetow
+
+Christmas:{time;#!!!
+
+Children: do tell $wishes;
+
+Santa: for $each (@children) { 
+BEGIN { read $each, $their, wishes and study them; use Memoize#ing
+
+} use constant gift, 'wrapping'; 
+package Gifts; pack $each, gift and bless $each and goto deliver
+or do import if not local $available,!!! HO, HO, HO;
+
+redo Santa, pipe $gifts, to_childs;
+redo Santa and do return if last one, is, delivered; 
+
+deliver: gift and require diagnostics if our $gifts ,not break;
+do{ use NEXT; time; tied $gifts} if broken and dump the, broken, ones;
+The_children: sleep and wait for (each %gift) and try { to => untie $gifts };
+
+redo Santa, pipe $gifts, to_childs;
+redo Santa and do return if last one, is, delivered; 
+
+The_christmas_tree: formline s/ /childrens/, $gifts;
+alarm and warn if not exists $Christmas{ tree}, @t, $ENV{HOME};  
+write <111
+ and send the
+EMail
+;wait and redo deliver until defined local $tree;
+
+redo Santa, pipe $gifts, to_childs;
+redo Santa and do return if last one, is, delivered ;}
+
+END {} our $mission and do sleep until next Christmas ;}
+
+__END__
+
+This is perl, v5.8.8 built for i386-freebsd-64int
+

+
+More Perl Poetry of mine
+
+

19. CSS3 is turing complete

+
+CSS3 is Turing complete because it can simulate a Turing machine using only CSS animations and styles without any JavaScript or external logic. This is achieved by using keyframe animations to change the styles of HTML elements in a way that encodes computation, performing calculations and state transitions.
+
+Is CSS turing complete?
+
+It is surprising because CSS is primarily a styling language intended for the presentation layer of web pages, not for computation or logic. Its capability to perform complex computations defies its typical use case and showcases the unintended computational power that can emerge from the creative use of seemingly straightforward technologies.
+
+Check out this 100% CSS implementation of the Conways Game of Life:
+
+
+
+CSS Conways Game of Life
+
+Conway's Game of Life is Turing complete because it can simulate a universal Turing machine, meaning it can perform any computation that a computer can, given the right initial conditions and sufficient time and space. Suppose a language can implement Conway's Game of Life. In that case, it demonstrates the language's ability to handle complex state transitions and computations. It has the necessary constructs (like iteration, conditionals, and data manipulation) to simulate any algorithm, thus confirming its Turing completeness.
+
+

20. The biggest shell programs

+
+One would think that shell scripts are only suitable for small tasks. Well, I must be wrong, as there are huge shell programs out there (up to 87k LOC) which aren't auto-generated but hand-written!
+
+The Biggest Sell Programs in the World
+
+My Gemtexter (bash) is only 1329 LOC as of now. So it's tiny.
+
+Gemtexter - One Bash script to rule it all
+
+I hope you had some fun. E-Mail your comments to paul@nospam.buetow.org :-)
+
+Back to the main site
+

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts

2025-01-30T09:22:06+02:00

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts

+
+Published at 2025-01-30T09:22:06+02:00
+
+This is the third blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution we will use on FreeBSD-based physical machines.
+
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts (You are currently reading this)
+
+
+
+

+
+

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
⇢ Introduction
⇢ Changes since last time
⇢ ⇢ FreeBSD upgrade from 14.1 to 14.2
⇢ ⇢ A new home (behind the TV)
⇢ The UPS hardware
⇢ Configuring FreeBSD to Work with the UPS
⇢ ⇢ USB Device Detection
⇢ ⇢ apcupsd Installation
⇢ ⇢ UPS Connectivity Test
⇢ APC Info on Partner Nodes:
⇢ ⇢ Installation on partners
⇢ Power outage simulation
⇢ ⇢ Pulling the plug
⇢ ⇢ Restoring power

Introduction

+
+In this blog post, we are setting up the UPS for the cluster. A UPS, or Uninterruptible Power Supply, safeguards my cluster from unexpected power outages and surges. It acts as a backup battery that kicks in when the electricity cuts out—especially useful in my area, where power cuts are frequent—allowing for a graceful system shutdown and preventing data loss and corruption. This is especially important since I will also store some of my data on the f3s nodes.
+
+

Changes since last time

+
+

FreeBSD upgrade from 14.1 to 14.2

+
+There has been a new release since the last blog post in this series. The upgrade from 14.1 was as easy as:
+
+ +

paul@f0: ~ % doas freebsd-update fetch
+paul@f0: ~ % doas freebsd-update install
+paul@f0: ~ % doas freebsd-update -r 14.2-RELEASE upgrade
+paul@f0: ~ % doas freebsd-update install
+paul@f0: ~ % doas shutdown -r now
+

+
+And after rebooting, I ran:
+
+ +

paul@f0: ~ % doas freebsd-update install
+paul@f0: ~ % doas pkg update
+paul@f0: ~ % doas pkg upgrade
+paul@f0: ~ % doas shutdown -r now
+

+
+And after another reboot, I was on 14.2:
+
+ +

paul@f0:~ % uname -a
+FreeBSD f0.lan.buetow.org 14.2-RELEASE FreeBSD 14.2-RELEASE 
+ releng/14.2-n269506-c8918d6c7412 GENERIC amd64
+

+
+And, of course, I ran this on all 3 nodes!
+
+

A new home (behind the TV)

+
+I've put all the infrastructure behind my TV, as plenty of space is available. The TV hides most of the setup, which drastically improved the SAF (spouse acceptance factor).
+
+
+
+I got rid of the mini-switch I mentioned in the previous blog post. I have the TP-Link EAP615-Wall mounted on the wall nearby, which is my OpenWrt-powered Wi-Fi hotspot. It also has 3 Ethernet ports, to which I connected the Beelink nodes. That's the device you see at the very top.
+
+The Ethernet cables go downward through the cable boxes to the Beelink nodes. In addition to the Beelink f3s nodes, I connected the TP-Link to the UPS as well (not discussed further in this blog post, but the positive side effect is that my Wi-Fi will still work during a power loss for some time—and during a power cut, the Beelink nodes will still be able to communicate with each other).
+
+On the very left (the black box) is the UPS, with four power outlets. Three go to the Beelink nodes, and one goes to the TP-Link. A USB output is also connected to the first Beelink node, f0.
+
+On the very right (halfway hidden behind the TV) are the 3 Beelink nodes stacked on top of each other. The only downside (or upside?) is that my 14-month-old daughter is now chaos-testing the Beelink nodes, as the red power buttons (now reachable for her) are very attractive for her to press when passing by randomly. :-) Luckily, that will only cause graceful system shutdowns!
+
+

The UPS hardware

+
+I wanted a UPS that I could connect to via FreeBSD, and that would provide enough backup power to operate the cluster for a couple of minutes (it turned out to be around an hour, but this time will likely be shortened after future hardware upgrades, like additional drives and a backup enclosure) and to automatically initiate the shutdown of all the f3s nodes.
+
+I decided on the APC Back-UPS BX750MI model because:
+
+

Zero noise level when there is no power cut (some light noise when the battery is in operation during a power cut).
Cost: It is relatively affordable (not costing thousands).
USB connectivity: Can be connected via USB to one of the FreeBSD hosts to read the UPS status.
A power output of 750VA (or 410 watts), suitable for an hour of runtime for my f3s nodes (plus the Wi-Fi router).
Multiple power outlets: Can connect all 3 f3s nodes directly.
User-replaceable batteries: I can replace the batteries myself after two years or more (depending on usage).
Its compact design. Overall, I like how it looks.

+
+
+

Configuring FreeBSD to Work with the UPS

+
+

USB Device Detection

+
+Once plugged in via USB on FreeBSD, I could see the following in the kernel messages:
+
+ +

paul@f0: ~ % doas dmesg | grep UPS
+ugen0.2:  at usbus0
+

+
+

apcupsd Installation

+
+To make use of the USB connection, the apcupsd package had to be installed:
+
+ +

paul@f0: ~ % doas install apcupsd
+

+
+I have made the following modifications to the configuration file so that the UPS can be used via the USB interface:
+
+ +

paul@f0:/usr/local/etc/apcupsd % diff -u apcupsd.conf.sample  apcupsd.conf
+--- apcupsd.conf.sample 2024-11-01 16:40:42.000000000 +0200
++++ apcupsd.conf        2024-12-03 10:58:24.009501000 +0200
+@@ -31,7 +31,7 @@
+ #     940-1524C, 940-0024G, 940-0095A, 940-0095B,
+ #     940-0095C, 940-0625A, M-04-02-2000
+ #
+-UPSCABLE smart
++UPSCABLE usb
+
+ # To get apcupsd to work, in addition to defining the cable
+ # above, you must also define a UPSTYPE, which corresponds to
+@@ -88,8 +88,10 @@
+ #                            that apcupsd binds to that particular unit
+ #                            (helpful if you have more than one USB UPS).
+ #
+-UPSTYPE apcsmart
+-DEVICE /dev/usv
++UPSTYPE usb
++DEVICE
+
+ # POLLTIME 
+ #   Interval (in seconds) at which apcupsd polls the UPS for status. This
+

+
+I left the remaining settings as the default ones; for example, the following are of main interest:
+
+

+# If during a power failure, the remaining battery percentage
+# (as reported by the UPS) is below or equal to BATTERYLEVEL,
+# apcupsd will initiate a system shutdown.
+BATTERYLEVEL 5
+
+# If during a power failure, the remaining runtime in minutes
+# (as calculated internally by the UPS) is below or equal to MINUTES,
+# apcupsd, will initiate a system shutdown.
+MINUTES 3
+

+
+I then enabled and started the daemon:
+
+ +

paul@f0:/usr/local/etc/apcupsd % doas sysrc apcupsd_enable=YES
+apcupsd_enable:  -> YES
+paul@f0:/usr/local/etc/apcupsd % doas service apcupsd start
+Starting apcupsd.
+

+
+

UPS Connectivity Test

+
+And voila, I could now access the UPS information via the apcaccess command; how convenient :-) (I also read through the manual page, which provides a good understanding of what else can be done with it!).
+
+ +

paul@f0:~ % apcaccess
+APC      : 001,035,0857
+DATE     : 2025-01-26 14:43:27 +0200
+HOSTNAME : f0.lan.buetow.org
+VERSION  : 3.14.14 (31 May 2016) freebsd
+UPSNAME  : f0.lan.buetow.org
+CABLE    : USB Cable
+DRIVER   : USB UPS Driver
+UPSMODE  : Stand Alone
+STARTTIME: 2025-01-26 14:43:25 +0200
+MODEL    : Back-UPS BX750MI
+STATUS   : ONLINE
+LINEV    : 230.0 Volts
+LOADPCT  : 4.0 Percent
+BCHARGE  : 100.0 Percent
+TIMELEFT : 65.3 Minutes
+MBATTCHG : 5 Percent
+MINTIMEL : 3 Minutes
+MAXTIME  : 0 Seconds
+SENSE    : Medium
+LOTRANS  : 145.0 Volts
+HITRANS  : 295.0 Volts
+ALARMDEL : No alarm
+BATTV    : 13.6 Volts
+LASTXFER : Automatic or explicit self test
+NUMXFERS : 0
+TONBATT  : 0 Seconds
+CUMONBATT: 0 Seconds
+XOFFBATT : N/A
+SELFTEST : NG
+STATFLAG : 0x05000008
+SERIALNO : 9B2414A03599
+BATTDATE : 2001-01-01
+NOMINV   : 230 Volts
+NOMBATTV : 12.0 Volts
+NOMPOWER : 410 Watts
+END APC  : 2025-01-26 14:44:06 +0200
+

+
+

APC Info on Partner Nodes:

+
+So far, so good. Host f0 would shut down itself when short on power. But what about the f1 and f2 nodes? They aren't connected directly to the UPS and, therefore, wouldn't know that their power is about to be cut off. For this, apcupsd running on the f1 and f2 nodes can be configured to retrieve UPS information via the network from the apcupsd server running on the f0 node, which is connected directly to the APC via USB.
+
+Of course, this won't work when f0 is down. In this case, no operational node would be connected to the UPS via USB; therefore, the current power status would not be known. However, I consider this a rare circumstance. Furthermore, in case of an f0 system crash, sudden power outages on the two other nodes would occur at different times making real data loss (the main concern here) less likely.
+
+And if f0 is down and f1 and f2 receive new data and crash midway, it's likely that a client (e.g., an Android app or another laptop) still has the data stored on it, making data recoverable and data loss overall nearly impossible. I'd receive an alert if any of the nodes go down (more on monitoring later in this blog series).
+
+

Installation on partners

+
+To do this, I installed apcupsd via doas pkg install apcupsd on f1 and f2, and then I could connect to it this way:
+
+ +

paul@f1:~ % apcaccess -h f0.lan.buetow.org | grep Percent
+LOADPCT  : 12.0 Percent
+BCHARGE  : 94.0 Percent
+MBATTCHG : 5 Percent
+

+
+But I want the daemon to be configured and enabled in such a way that it connects to the master UPS node (the one with the UPS connected via USB) so that it can also initiate a system shutdown when the UPS battery reaches low levels. For that, apcupsd itself needs to be aware of the UPS status.
+
+On f1 and f2, I changed the configuration to use f0 (where apcupsd is listening) as a remote device. I also changed the MINUTES setting from 3 to 6 and the BATTERYLEVEL setting from 5 to 10 to ensure that the f1 and f2 nodes could still connect to the f0 node for UPS information before f0 decides to shut down itself. So f1 and f2 must shut down earlier than f0:
+
+ +

paul@f2:/usr/local/etc/apcupsd % diff -u apcupsd.conf.sample apcupsd.conf
+--- apcupsd.conf.sample 2024-11-01 16:40:42.000000000 +0200
++++ apcupsd.conf        2025-01-26 15:52:45.108469000 +0200
+@@ -31,7 +31,7 @@
+ #     940-1524C, 940-0024G, 940-0095A, 940-0095B,
+ #     940-0095C, 940-0625A, M-04-02-2000
+ #
+-UPSCABLE smart
++UPSCABLE ether
+
+ # To get apcupsd to work, in addition to defining the cable
+ # above, you must also define a UPSTYPE, which corresponds to
+@@ -52,7 +52,6 @@
+ #                            Network Information Server. This is used if the
+ #                            UPS powering your computer is connected to a
+ #                            different computer for monitoring.
+-#
+ # snmp      hostname:port:vendor:community
+ #                            SNMP network link to an SNMP-enabled UPS device.
+ #                            Hostname is the ip address or hostname of the UPS
+@@ -88,8 +87,8 @@
+ #                            that apcupsd binds to that particular unit
+ #                            (helpful if you have more than one USB UPS).
+ #
+-UPSTYPE apcsmart
+-DEVICE /dev/usv
++UPSTYPE net
++DEVICE f0.lan.buetow.org:3551
+
+ # POLLTIME 
+ #   Interval (in seconds) at which apcupsd polls the UPS for status. This
+@@ -147,12 +146,12 @@
+ # If during a power failure, the remaining battery percentage
+ # (as reported by the UPS) is below or equal to BATTERYLEVEL,
+ # apcupsd will initiate a system shutdown.
+-BATTERYLEVEL 5
++BATTERYLEVEL 10
+
+ # If during a power failure, the remaining runtime in minutes
+ # (as calculated internally by the UPS) is below or equal to MINUTES,
+ # apcupsd, will initiate a system shutdown.
+-MINUTES 3
++MINUTES 6
+
+ # If during a power failure, the UPS has run on batteries for TIMEOUT
+ # many seconds or longer, apcupsd will initiate a system shutdown.
+
+

+So I also ran the following commands on f1 and f2:
+
+ +

paul@f1:/usr/local/etc/apcupsd % doas sysrc apcupsd_enable=YES
+apcupsd_enable:  -> YES
+paul@f1:/usr/local/etc/apcupsd % doas service apcupsd start
+Starting apcupsd.
+

+
+And then I was able to connect to localhost via the apcaccess command:
+
+ +

paul@f1:~ % doas apcaccess | grep Percent
+LOADPCT  : 5.0 Percent
+BCHARGE  : 95.0 Percent
+MBATTCHG : 5 Percent
+

+
+

Power outage simulation

+
+

Pulling the plug

+
+I simulated a power outage by removing the power input from the APC. Immediately, the following message appeared on all the nodes:
+
+

+Broadcast Message from root@f0.lan.buetow.org
+        (no tty) at 15:03 EET...
+
+Power failure. Running on UPS batteries.                                              
+

+
+I ran the following command to confirm the available battery time:
+
+ +

paul@f0:/usr/local/etc/apcupsd % apcaccess -p TIMELEFT
+63.9 Minutes
+

+
+And after around one hour (f1 and f2 a bit earlier, f0 a bit later due to the different BATTERYLEVEL and MINUTES settings outlined earlier), the following broadcast was sent out:
+
+

+Broadcast Message from root@f0.lan.buetow.org
+        (no tty) at 15:08 EET...
+
+        *** FINAL System shutdown message from root@f0.lan.buetow.org ***
+
+System going down IMMEDIATELY
+
+apcupsd initiated shutdown
+

+
+And all the nodes shut down safely before the UPS ran out of battery!
+
+

Restoring power

+
+After restoring power, I checked the logs in /var/log/daemon.log and found the following on all 3 nodes:
+
+

+Jan 26 17:36:24 f2 apcupsd[2159]: Power failure.
+Jan 26 17:36:30 f2 apcupsd[2159]: Running on UPS batteries.
+Jan 26 17:36:30 f2 apcupsd[2159]: Battery charge below low limit.
+Jan 26 17:36:30 f2 apcupsd[2159]: Initiating system shutdown!
+Jan 26 17:36:30 f2 apcupsd[2159]: User logins prohibited
+Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd exiting, signal 15
+Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded
+

+
+All good :-) See you in the next post of this series!
+
+Other BSD related posts are:
+
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts (You are currently reading this)
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-04-01 KISS high-availability with OpenBSD
+2024-01-13 One reason why I love OpenBSD
+2022-10-30 Installing DTail on OpenBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Back to the main site
+

Working with an SRE Interview

2025-01-15T00:16:04+02:00

Working with an SRE Interview

+
+Published at 2025-01-15T00:16:04+02:00
+
+I have been interviewed by Florian Buetow on cracking-ai-engineering.com about what it's like working with a Site Reliability Engineer from the point of view of a Software Engineer, Data Scientist, and AI Engineer.
+
+See original interview here
+Cracking AI Engineering
+
+Below, I am posting the interview here on my blog as well.
+
+

+
+

Working with an SRE Interview
⇢ Preamble
⇢ Introducing Paul
⇢ How did you get started?
⇢ Roles and Career Progression
⇢ Anecdotes and Best Practices
⇢ Working with Different Teams
⇢ Using AI Tools
⇢ SRE Learning Resources
⇢ Blogging
⇢ Wrap-up
⇢ Closing comments

Preamble

+
+In this insightful interview, Paul Bütow, a Principal Site Reliability Engineer at Mimecast, shares over a decade of experience in the field. Paul highlights the role of an Embedded SRE, emphasizing the importance of automation, observability, and effective incident management. We also focused on the key question of how you can work effectively with an SRE weather you are an individual contributor or a manager, a software engineer or data scientist. And how you can learn more about site reliability engineering.
+
+

Introducing Paul

+
+Hi Paul, please introduce yourself briefly to the audience. Who are you, what do you do for a living, and where do you work?
+
+My name is Paul Bütow, I work at Mimecast, and I’m a Principal Site Reliability Engineer there. I’ve been with Mimecast for almost ten years now. The company specializes in email security, including things like archiving, phishing detection, malware protection, and spam filtering.
+
+You mentioned that you’re an ‘Embedded SRE.’ What does that mean exactly?
+
+It means that I’m directly part of the software engineering team, not in a separate Ops department. I ensure that nothing is deployed manually, and everything runs through automation. I also set up monitoring and observability. These are two distinct aspects: monitoring alerts us when something breaks, while observability helps us identify trends. I also create runbooks so we know what to do when specific incidents occur frequently.
+
+Infrastructure SREs on the other hand handle the foundational setup, like providing the Kubernetes cluster itself or ensuring the operating systems are installed. They don't work on the application directly but ensure the base infrastructure is there for others to use. This works well when a company has multiple teams that need shared infrastructure.
+
+

How did you get started?

+
+How did your interest in Linux or FreeBSD start?
+
+It began during my school days. We had a PC with DOS at home, and I eventually bought Suse Linux 5.3. Shortly after, I discovered FreeBSD because I liked its handbook so much. I wanted to understand exactly how everything worked, so I also tried Linux from Scratch. That involves installing every package manually to gain a better understanding of operating systems.
+
+https://www.FreeBSD.org
+https://linuxfromscratch.org/
+
+And after school, you pursued computer science, correct?
+
+Exactly. I wasn’t sure at first whether I wanted to be a software developer or a system administrator. I applied for both and eventually accepted an offer as a Linux system administrator. This was before 'SRE' became a buzzword, but much of what I did back then-automation, infrastructure as code, monitoring-is now considered part of the typical SRE role.
+
+

Roles and Career Progression

+
+Tell us about how you joined Mimecast. When did you fully embrace the SRE role?
+
+I started as a Linux sysadmin at 1&1. I managed an ad server farm with hundreds of systems and later handled load balancers. Together with an architect, we managed F5 load balancers distributing around 2,000 services, including for portals like web.de and GMX. I also led the operations team technically for a while before moving to London to join Mimecast.
+
+At Mimecast, the job title was explicitly 'Site Reliability Engineer.' The biggest difference was that I was no longer in a separate Ops department but embedded directly within the storage and search backend team. I loved that because we could plan features together-from automation to measurability and observability. Mimecast also operates thousands of physical servers for email archiving, which was fascinating since I already had experience with large distributed systems at 1&1. It was the right step for me because it allowed me to work close to the code while remaining hands-on with infrastructure.
+
+What are the differences between SRE, DevOps, SysAdmin, and Architects?
+
+SREs are like the next step after SysAdmins. A SysAdmin might manually install servers, replace disks, or use simple scripts for automation, while SREs use infrastructure as code and focus on reliability through SLIs, SLOs, and automation. DevOps isn’t really a job-it’s more of a way of working, where developers are involved in operations tasks like setting up CI/CD pipelines or on-call shifts. Architects focus on designing systems and infrastructures, such as load balancers or distributed systems, working alongside SREs to ensure the systems meet the reliability and scalability requirements. The specific responsibilities of each role depend on the company, and there is often overlap.
+
+What are the most important reliability lessons you’ve learned so far?
+
+

Don’t leave SRE aspects as an afterthought. It’s much better to discuss automation, monitoring, SLIs, and SLOs early on. Traditional sysadmins often installed systems manually, but today, we do everything via infrastructure as code-using tools like Terraform or Puppet.
I also distinguish between monitoring and observability. Monitoring tells us, 'The server is down, alarm!' Observability dives deeper, showing trends like increasing latency so we can act proactively.
SLI, SLO, and SLA are core elements. We focus on what users actually experience-for example, how quickly an email is sent-and set our goals accordingly.
Runbooks are also crucial. When something goes wrong at night, you don’t want to start from scratch. A runbook outlines how to debug and resolve specific problems, saving time and reducing downtime.

Anecdotes and Best Practices

+
+Runbooks sound very practical. Can you explain how they’re used day-to-day?
+
+Runbooks are essentially guides for handling specific incidents. For instance, if a service won’t start, the runbook will specify where the logs are and which commands to use. Observability takes it a step further, helping us spot changes early-like rising error rates or latency-so we can address issues before they escalate.
+
+When should you decide to put something into a runbook, and when is it unnecessary?
+
+If an issue happens frequently, it should be documented in a runbook so that anyone, even someone new, can follow the steps to fix it. The idea is that 90% of the common incidents should be covered. For example, if a service is down, the runbook would specify where to find logs, which commands to check, and what actions to take. On the other hand, rare or complex issues, where the resolution depends heavily on context or varies each time, don’t make sense to include in detail. For those, it’s better to focus on general troubleshooting steps.
+
+How do you search for and find the correct runbooks?
+
+Runbooks should be linked directly in the alert you receive. For example, if you get an alert about a service not running, the alert will have a link to the runbook that tells you what to check, like logs or commands to run. Runbooks are best stored in an internal wiki, so if you don’t find the link in the alert, you know where to search. The important thing is that runbooks are easy to find and up to date because that’s what makes them useful during incidents.
+
+Do you have an interesting war story you can share with us?
+
+Sure. At 1&1, we had a proprietary ad server software that ran a SQL query during startup. The query got slower over time, eventually timing out and preventing the server from starting. Since we couldn’t access the source code, we searched the binary for the SQL and patched it. By pinpointing the issue, a developer was able to adjust the SQL. This collaboration between sysadmin and developer perspectives highlights the value of SRE work.
+
+

Working with Different Teams

+
+You’re embedded in a team-how does collaboration with developers work practically?
+
+We plan everything together from the start. If there’s a new feature, we discuss infrastructure, automated deployments, and monitoring right away. Developers are experts in the code, and I bring the infrastructure expertise. This avoids unpleasant surprises before going live.
+
+How about working with data scientists or ML engineers? Are there differences?
+
+The principles are the same. ML models also need to be deployed and monitored. You deal with monitoring, resource allocation, and identifying performance drops. Whether it’s a microservice or an ML job, at the end of the day, it’s all running on servers or clusters that must remain stable.
+
+What about working with managers or the FinOps team?
+
+We often discuss costs, especially in the cloud, where scaling up resources is easy. It’s crucial to know our metrics: do we have enough capacity? Do we need all instances? Or is the CPU only at 5% utilization? This data helps managers decide whether the budget is sufficient or if optimizations are needed.
+
+Do you have practical tips for working with SREs?
+
+Yes, I have a few:
+
+

Early involvement: Include SREs from the beginning in your project.
Runbooks & documentation: Document recurring errors.
Try first: Try to understand the issue yourself before immediately asking the SRE.
Basic infra knowledge: Kubernetes and Terraform aren’t magic. Some basic understanding helps every developer.

Using AI Tools

+
+Let’s talk about AI. How do you use it in your daily work?
+
+For boilerplate code, like Terraform snippets, I often use ChatGPT. It saves time, although I always review and adjust the output. Log analysis is another exciting application. Instead of manually going through millions of lines, AI can summarize key outliers or errors.
+
+Do you think AI could largely replace SREs or significantly change the role?
+
+I see AI as an additional tool. SRE requires a deep understanding of how distributed systems work internally. While AI can assist with routine tasks or quickly detect anomalies, human expertise is indispensable for complex issues.
+
+

SRE Learning Resources

+
+What resources would you recommend for learning about SRE?
+
+The Google SRE book is a classic, though a bit dry. I really like 'Seeking SRE,' as it offers various perspectives on SRE, with many practical stories from different companies.
+
+https://sre.google/books/
+Seeking SRE
+
+Do you have a podcast recommendation?
+
+The Google SRE prodcast is quite interesting. It offers insights into how Google approaches SRE, along with perspectives from external guests.
+
+https://sre.google/prodcast/
+
+

Blogging

+
+You also have a blog. What motivates you to write regularly?
+
+Writing helps me learn the most. It also serves as a personal reference. Sometimes I look up how I solved a problem a year ago. And of course, others tackling similar projects might find inspiration in my posts.
+
+What do you blog about?
+
+Mostly technical topics I find exciting, like homelab projects, Kubernetes, or book summaries on IT and productivity. It’s a personal blog, so I write about what I enjoy.
+
+

Wrap-up

+
+To wrap up, what are three things every team should keep in mind for stability?
+
+First, maintain runbooks and documentation to avoid chaos at night. Second, automate everything-manual installs in production are risky. Third, define SLIs, SLOs, and SLAs early so everyone knows what we’re monitoring and guaranteeing.
+
+Is there a motto or mindset that particularly inspires you as an SRE?
+
+"Keep it simple and stupid"-KISS. Not everything has to be overly complex. And always stay curious. I’m still fascinated by how systems work under the hood.
+
+Where can people find you online?
+
+You can find links to my socials on my website paul.buetow.org
+I regularly post articles and link to everything else I’m working on outside of work.
+
+https://paul.buetow.org
+
+Thank you very much for your time and this insightful interview into the world of site reliability engineering
+
+My pleasure, this was fun.
+
+

Closing comments

+
+Dear reader, I hope this conversation with Paul Bütow provided an exciting peak into the world of Site Reliability Engineering. Whether you’re a software developer, data scientist, ML engineer, or manager, reliable systems are always a team effort. Hopefully, you’ve taken some insights or tips from Paul’s experiences for your own team or next project. Thanks for joining us, and best of luck refining your own SRE practices!
+
+E-Mail your comments to paul@nospam.buetow.org or contact Florian via the Cracking AI Engineering :-)
+
+Back to the main site
+

Posts from October to December 2024

2024-12-31T18:09:58+02:00

Posts from October to December 2024

+
+Published at 2024-12-31T18:09:58+02:00
+
+Happy new year!
+
+These are my social media posts from the last three months. I keep them here to reflect on them and also to not lose them. Social media networks come and go and are not under my control, but my domain is here to stay.
+
+These are from Mastodon and LinkedIn. Have a look at my about page for my social media profiles. This list is generated with Gos, my social media platform sharing tool.
+
+My about page
+https://codeberg.org/snonux/gos
+
+

+
+

Posts from October to December 2024
Posts for 202410 202411 202412
⇢ October 2024
⇢ ⇢ First on-call experience in a startup. Doesn't ...
⇢ ⇢ Reviewing your own PR or MR before asking ...
⇢ ⇢ Fun with defer in #golang, I did't know, that ...
⇢ ⇢ I have been in incidents. Understandably, ...
⇢ ⇢ Little tips using strings in #golang and I ...
⇢ ⇢ Reading this post about #rust (especially the ...
⇢ ⇢ The opposite of #ChaosMonkey ... ...
⇢ November 2024
⇢ ⇢ I just became a Silver Patreon for OSnews. What ...
⇢ ⇢ Until now, I wasn't aware, that Go is under a ...
⇢ ⇢ These are some book notes from "Staff Engineer" ...
⇢ ⇢ Looking at #Kubernetes, it's pretty much ...
⇢ ⇢ There has been an outage at the upstream ...
⇢ ⇢ One of the more confusing parts in Go, nil ...
⇢ ⇢ Agreeably, writing down with Diagrams helps you ...
⇢ ⇢ I like the idea of types in Ruby. Raku is ...
⇢ ⇢ So, #Haskell is better suited for general ...
⇢ ⇢ At first, functional options add a bit of ...
⇢ ⇢ Revamping my home lab a little bit. #freebsd ...
⇢ ⇢ Wondering to which #web #browser I should ...
⇢ ⇢ eks-node-viewer is a nifty tool, showing the ...
⇢ ⇢ Have put more Photos on - On my static photo ...
⇢ ⇢ In Go, passing pointers are not automatically ...
⇢ ⇢ Myself being part of an on-call rotations over ...
⇢ ⇢ Feels good to code in my old love #Perl again ...
⇢ ⇢ This is an interactive summary of the Go ...
⇢ December 2024
⇢ ⇢ Thats unexpected, you cant remove a NaN key ...
⇢ ⇢ My second blog post about revamping my home lab ...
⇢ ⇢ Very insightful article about tech hiring in ...
⇢ ⇢ for #bpf #ebpf performance debugging, have ...
⇢ ⇢ 89 things he/she knows about Git commits is a ...
⇢ ⇢ I found that working on multiple side projects ...
⇢ ⇢ Agreed? Agreed. Besides #Ruby, I would also ...
⇢ ⇢ Plan9 assembly format in Go, but wait, it's not ...
⇢ ⇢ This is a neat blog post about the Helix text ...
⇢ ⇢ This blog post is basically a rant against ...
⇢ ⇢ Quick trick to get Helix themes selected ...
⇢ ⇢ Example where complexity attacks you from ...
⇢ ⇢ LLMs for Ops? Summaries of logs, probabilities ...
⇢ ⇢ Excellent article about your dream Product ...
⇢ ⇢ I just finished reading all chapters of CPU ...
⇢ ⇢ Indeed, useful to know this stuff! #sre ...
⇢ ⇢ It's the small things, which make Unix like ...
⇢ ⇢ My New Year's resolution is not to start any ...

Posts for 202410 202411 202412

+
+

October 2024

+
+

First on-call experience in a startup. Doesn't ...

+
+First on-call experience in a startup. Doesn't sound a lot of fun! But the lessons were learned! #sre
+
+ntietz.com/blog/lessons-from-my-first-on-call/
+
+

Reviewing your own PR or MR before asking ...

+
+Reviewing your own PR or MR before asking others to review it makes a lot of sense. Have seen so many silly mistakes which would have been avoided. Saving time for the real reviewer.
+
+www.jvt.me/posts/2019/01/12/self-code-review/
+
+

Fun with defer in #golang, I did't know, that ...

+
+Fun with defer in #golang, I did't know, that a defer object can either be heap or stack allocated. And there are some rules for inlining, too.
+
+victoriametrics.com/blog/defer-in-go/
+
+

I have been in incidents. Understandably, ...

+
+I have been in incidents. Understandably, everyone wants the issue to be resolved as quickly and others want to know how long TTR will be. IMHO, providing no estimates at all is no solution either. So maybe give a rough estimate but clearly communicate that the estimate is rough and that X, Y, and Z can interfere, meaning there is a chance it will take longer to resolve the incident. Just my thought. What's yours?
+
+firehydrant.com/blog/hot-take-dont-provide-incident-resolution-estimates/
+
+

Little tips using strings in #golang and I ...

+
+Little tips using strings in #golang and I personally think one must look more into the std lib (not just for strings, also for slices, maps,...), there are tons of useful helper functions.
+
+www.calhoun.io/6-tips-for-using-strings-in-go/
+
+

Reading this post about #rust (especially the ...

+
+Reading this post about #rust (especially the first part), I think I made a good choice in deciding to dive into #golang instead. There was a point where I wanted to learn a new programming language, and Rust was on my list of choices. I think the Go project does a much better job of deciding what goes into the language and how. What are your thoughts?
+
+josephg.com/blog/rewriting-rust/
+
+

The opposite of #ChaosMonkey ... ...

+
+The opposite of #ChaosMonkey ... automatically repairing and healing services helping to reduce manual toil work. Runbooks and scripts are only the first step, followed by a fully blown service written in Go. Could be useful, but IMHO why not rather address the root causes of the manual toil work? #sre
+
+blog.cloudflare.com/nl-nl/improving-platform-resilience-at-cloudflare/
+
+

November 2024

+
+

I just became a Silver Patreon for OSnews. What ...

+
+I just became a Silver Patreon for OSnews. What is OSnews? It is an independent news site about IT. It is slightly independent and, at times, alternative. I have enjoyed it since my early student days. This one and other projects I financially support are listed here:
+
+foo.zone/gemfeed/2024-09-07-projects-i-support.gmi (Gemini)
+foo.zone/gemfeed/2024-09-07-projects-i-support.html
+
+

Until now, I wasn't aware, that Go is under a ...

+
+Until now, I wasn't aware, that Go is under a BSD-style license (3-clause as it seems). Neat. I don't know why, but I always was under the impression it would be MIT. #bsd #golang
+
+go.dev/LICENSE
+
+

These are some book notes from "Staff Engineer" ...

+
+These are some book notes from "Staff Engineer" – there is some really good insight into what is expected from a Staff Engineer and beyond in the industry. I wish I had read the book earlier.
+
+foo.zone/gemfeed/2024-10-24-staff-engineer-book-notes.gmi (Gemini)
+foo.zone/gemfeed/2024-10-24-staff-engineer-book-notes.html
+
+

Looking at #Kubernetes, it's pretty much ...

+
+Looking at #Kubernetes, it's pretty much following the Unix way of doing things. It has many tools, but each tool has its own single purpose: DNS, scheduling, container runtime, various controllers, networking, observability, alerting, and more services in the control plane. Everything is managed by different services or plugins, mostly running in their dedicated pods. They don't communicate through pipes, but network sockets, though. #k8s
+
+

There has been an outage at the upstream ...

+
+There has been an outage at the upstream network provider for OpenBSD.Amsterdam (hoster, I am using). This was the first real-world test for my KISS HA setup, and it worked flawlessly! All my sites and services failed over automatically to my other #OpenBSD VM!
+
+foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi (Gemini)
+foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html
+openbsd.amsterdam/
+
+

One of the more confusing parts in Go, nil ...

+
+One of the more confusing parts in Go, nil values vs nil errors: #golang
+
+unexpected-go.com/nil-errors-that-are-non-nil-errors.html
+
+

Agreeably, writing down with Diagrams helps you ...

+
+Agreeably, writing down with Diagrams helps you to think things more through. And keeps others on the same page. Only worth for projects from a certain size, IMHO.
+
+ntietz.com/blog/reasons-to-write-design-docs/
+
+

I like the idea of types in Ruby. Raku is ...

+
+I like the idea of types in Ruby. Raku is supports that already, but in Ruby, you must specify the types in a separate .rbs file, which is, in my opinion, cumbersome and is a reason not to use it extensively for now. I believe there are efforts to embed the type information in the standard .rb files, and that the .rbs is just an experiment to see how types could work out without introducing changes into the core Ruby language itself right now? #Ruby #RakuLang
+
+github.com/ruby/rbs
+
+

So, #Haskell is better suited for general ...

+
+So, #Haskell is better suited for general purpose than #Rust? I thought deploying something in Haskell means publishing an academic paper :-) Interesting rant about Rust, though:
+
+chrisdone.com/posts/rust/
+
+

At first, functional options add a bit of ...

+
+At first, functional options add a bit of boilerplate, but they turn out to be quite neat, especially when you have very long parameter lists that need to be made neat and tidy. #golang
+
+www.calhoun.io/using-functional-options-instead-of-method-chaining-in-go/
+
+

Revamping my home lab a little bit. #freebsd ...

+
+Revamping my home lab a little bit. #freebsd #bhyve #rocky #linux #vm #k3s #kubernetes #wireguard #zfs #nfs #ha #relayd #k8s #selfhosting #homelab
+
+foo.zone/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi (Gemini)
+foo.zone/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.html
+
+

Wondering to which #web #browser I should ...

+
+Wondering to which #web #browser I should switch now personally ...
+
+www.osnews.com/story/141100/mozilla-fo..-..dvocacy-for-open-web-privacy-and-more/
+
+

eks-node-viewer is a nifty tool, showing the ...

+
+eks-node-viewer is a nifty tool, showing the compute nodes currently in use in the #EKS cluster. especially useful when dynamically allocating nodes with #karpenter or auto scaling groups.
+
+github.com/awslabs/eks-node-viewer
+
+

Have put more Photos on - On my static photo ...

+
+Have put more Photos on - On my static photo sites - Generated with a #bash script
+
+irregular.ninja
+
+

In Go, passing pointers are not automatically ...

+
+In Go, passing pointers are not automatically faster than values. Pointers often force the memory to be allocated on the heap, adding GC overhad. With values, Go can determine whether to put the memory on the stack instead. But with large structs/objects (how you want to call them) or if you want to modify state, then pointers are the semantic to use. #golang
+
+blog.boot.dev/golang/pointers-faster-than-values/
+
+

Myself being part of an on-call rotations over ...

+
+Myself being part of an on-call rotations over my whole professional life, just have learned this lesson "Tell people who are new to on-call: Just have fun" :-) This is a neat blog post to read:
+
+ntietz.com/blog/what-i-tell-people-new-to-oncall/
+
+

Feels good to code in my old love #Perl again ...

+
+Feels good to code in my old love #Perl again after a while. I am implementing a log parser for generating site stats of my personal homepage! :-) @Perl
+
+

This is an interactive summary of the Go ...

+
+This is an interactive summary of the Go release, with a lot of examples utilising iterators in the slices and map packages. Love it! #golang
+
+antonz.org/go-1-23/
+
+

December 2024

+
+

Thats unexpected, you cant remove a NaN key ...

+
+Thats unexpected, you cant remove a NaN key from a map without clearing it! #golang
+
+unexpected-go.com/you-cant-remove-a-nan-key-from-a-map-without-clearing-it.html
+
+

My second blog post about revamping my home lab ...

+
+My second blog post about revamping my home lab a little bit just hit the net. #FreeBSD #ZFS #n100 #k8s #k3s #kubernetes
+
+foo.zone/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi (Gemini)
+foo.zone/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.html
+
+

Very insightful article about tech hiring in ...

+
+Very insightful article about tech hiring in the age of LLMs. As an interviewer, I have experienced some of the scrnarios already first hand...
+
+newsletter.pragmaticengineer.com/p/how-genai-changes-tech-hiring
+
+

for #bpf #ebpf performance debugging, have ...

+
+for #bpf #ebpf performance debugging, have a look at bpftop from Netflix. A neat tool showing you the estimated CPU time and other performance statistics for all the BPF programs currently loaded into the #linux kernel. Highly recommend!
+
+github.com/Netflix/bpftop
+
+

89 things he/she knows about Git commits is a ...

+
+89 things he/she knows about Git commits is a neat list of #Git wisdoms
+
+www.jvt.me/posts/2024/07/12/things-know-commits/
+
+

I found that working on multiple side projects ...

+
+I found that working on multiple side projects concurrently is better than concentrating on just one. This seems inefficient at first, but whenever you tend to lose motivation, you can temporarily switch to another one with full élan. However, remember to stop starting and start finishing. This doesn't mean you should be working on 10+ (and a growing list of) side projects concurrently! Select your projects and commit to finishing them before starting the next thing. For example, my current limit of concurrent side projects is around five.
+
+

Agreed? Agreed. Besides #Ruby, I would also ...

+
+Agreed? Agreed. Besides #Ruby, I would also add #RakuLang and #Perl @Perl to the list of languages that are great for shell scripts - "Making Easy Things Easy and Hard Things Possible"
+
+lucasoshiro.github.io/posts-en/2024-06-17-ruby-shellscript/
+
+

Plan9 assembly format in Go, but wait, it's not ...

+
+Plan9 assembly format in Go, but wait, it's not the Operating System Plan9! #golang #rabbithole
+
+www.osnews.com/story/140941/go-plan9-memo-speeding-up-calculations-450/
+
+

This is a neat blog post about the Helix text ...

+
+This is a neat blog post about the Helix text editor, to which I personally switched around a year ago (from NeoVim). I should blog about my experience as well. To summarize: I am using it together with the terminal multiplexer #tmux. It doesn't bother me that Helix is purely terminal-based and therefore everything has to be in the same font. #HelixEditor
+
+jonathan-frere.com/posts/helix/
+
+

This blog post is basically a rant against ...

+
+This blog post is basically a rant against DataDog... Personally, I don't have much experience with DataDog (actually, I have never used it), but one reason to work with logs at my day job (with over 2,000 physical server machines) and to be cost-effective is by using dtail! #dtail #logs #logmanagement
+
+crys.site/blog/2024/reinventint-the-weel/
+dtail.dev
+
+

Quick trick to get Helix themes selected ...

+
+Quick trick to get Helix themes selected randomly #HelixEditor
+
+foo.zone/gemfeed/2024-12-15-random-helix-themes.gmi (Gemini)
+foo.zone/gemfeed/2024-12-15-random-helix-themes.html
+
+

Example where complexity attacks you from ...

+
+Example where complexity attacks you from behind #k8s #kubernetes #OpenAI
+
+surfingcomplexity.blog/2024/12/14/quic..-..ecent-openai-public-incident-write-up/
+
+

LLMs for Ops? Summaries of logs, probabilities ...

+
+LLMs for Ops? Summaries of logs, probabilities about correctness, auto-generating Ansible, some uses cases are there. Wouldn't trust it fully, though.
+
+youtu.be/WodaffxVq-E?si=noY0egrfl5izCSQI
+
+

Excellent article about your dream Product ...

+
+Excellent article about your dream Product Manager: Why every software team needs a product manager to thrive via @wallabagapp
+
+testdouble.com/insights/why-product-ma..-..s-accelerate-improve-software-delivery
+
+

I just finished reading all chapters of CPU ...

+
+I just finished reading all chapters of CPU land: ... not claiming to remember every detail, but it is a great refresher how CPUs and operating systems actually work under the hood when you execute a program, which we tend to forget in our higher abstraction world. I liked the "story" and some of the jokes along the way! Size wise, it is pretty digestable (not talking about books, but only 7 web articles/chapters)! #cpu #linux #unix #kernel #macOS
+
+cpu.land/
+
+

Indeed, useful to know this stuff! #sre ...

+
+Indeed, useful to know this stuff! #sre
+
+biriukov.dev/docs/resolver-dual-stack-..-..resolvers-and-dual-stack-applications/
+
+

It's the small things, which make Unix like ...

+
+It's the small things, which make Unix like systems, like GNU/Linux, interesting. Didn't know about this #GNU #Tar behaviour yet:
+
+xeiaso.net/notes/2024/pop-quiz-tar/
+
+

My New Year's resolution is not to start any ...

+
+My New Year's resolution is not to start any new non-fiction books (or only very few) but to re-read and listen to my favorites, which I read to reflect on and see things from different perspectives. Every time you re-read a book, you gain new insights.17491
+
+Other related posts:
+
+2025-01-01 Posts from October to December 2024 (You are currently reading this)
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Back to the main site
+

Random Weird Things - Part Ⅱ

2025-02-08T11:06:16+02:00

Random Weird Things - Part Ⅱ

-
-Published at 2025-02-08T11:06:16+02:00
-
-Every so often, I come across random, weird, and unexpected things on the internet. I thought it would be neat to share them here from time to time. This is the second run.
-
-2024-07-05 Random Weird Things - Part Ⅰ
-2025-02-08 Random Weird Things - Part Ⅱ (You are currently reading this)
-
-

-/\_/\           /\_/\
-( o.o ) WHOA!! ( o.o )
-> ^ <           > ^ <
-/   \    MOEEW! /   \
-/______\       /______\
-

-
-

Random Weird Things - Part Ⅱ
⇢ 11. The SQLite codebase is a gem
⇢ Go Programming
⇢ ⇢ 12. Official Go font
⇢ ⇢ 13. Go functions can have methods
⇢ macOS
⇢ ⇢ 14. ß and ss are treated the same
⇢ ⇢ 15. Colon as file path separator
⇢ 16. Polyglots - programs written in multiple languages
⇢ 17. Languages, where indices start at 1
⇢ 18. Perl Poetry
⇢ 19. CSS3 is turing complete
⇢ 20. The biggest shell programs

11. The SQLite codebase is a gem

-
-Check this out:
-
-
-
-Source:
-
-https://wetdry.world/@memes/112717700557038278
-
-

Go Programming

-
-

12. Official Go font

-
-The Go programming language has an official font called "Go Font." It was created to complement the aesthetic of the Go language, ensuring clear and legible rendering of code. The font includes a monospace version for code and a proportional version for general text, supporting consistent look and readability in Go-related materials and development environments.
-
-Check out some Go code displayed using the Go font:
-
-
-
-https://go.dev/blog/go-fonts
-
-The design emphasizes simplicity and readability, reflecting Go's philosophy of clarity and efficiency.
-
-I found it interesting and/or weird, as Go is a programming language. Why should it bother having its own font? I have never seen another open-source project like Go do this. But I also like it. Maybe I will use it in the future for this blog :-)
-
-

13. Go functions can have methods

-
-Functions on struct types? Well, know. Functions on types like int and string? It's also known of, but a bit lesser. Functions on function types? That sounds a bit funky, but it's possible, too! For demonstration, have a look at this snippet:
-
- -

package main
-
-import "log"
-
-type fun func() string
-
-func (f fun) Bar() string {
-        return "Bar"
-}
-
-func main() {
-        var f fun = func() string {
-                return "Foo"
-        }
-        log.Println("Example 1: ", f())
-        log.Println("Example 2: ", f.Bar())
-        log.Println("Example 3: ", fun(f.Bar).Bar())
-        log.Println("Example 4: ", fun(fun(f.Bar).Bar).Bar())
-}
-

-
-It runs just fine:
-
- -

❯ go run main.go
-2025/02/07 22:56:14 Example 1:  Foo
-2025/02/07 22:56:14 Example 2:  Bar
-2025/02/07 22:56:14 Example 3:  Bar
-2025/02/07 22:56:14 Example 4:  Bar
-

-
-

macOS

-
-For personal computing, I don't use Apple, but I have to use it for work.
-
-

14. ß and ss are treated the same

-
-Know German? In German, the letter "sarp s" is written as ß. ß is treated the same as ss on macOS.
-
-On a case-insensitive file system like macOS, not only are uppercase and lowercase letters treated the same, but non-Latin characters like the German "ß" are also considered equivalent to their Latin counterparts (in this case, "ss").
-
-So, even though "Maß" and "Mass" are not strictly equivalent, the macOS file system still treats them as the same filename due to its handling of Unicode characters. This can sometimes lead to unexpected behaviour. Check this out:
-
- -

❯ touch Maß
-❯ ls -l
--rw-r--r--@ 1 paul  wheel  0 Feb  7 23:02 Maß
-❯ touch Mass
-❯ ls -l
--rw-r--r--@ 1 paul  wheel  0 Feb  7 23:02 Maß
-❯ rm Mass
-❯ ls -l
-
-❯ touch Mass
-❯ ls -ltr
--rw-r--r--@ 1 paul  wheel  0 Feb  7 23:02 Mass
-❯ rm Maß
-❯ ls -l
-
-

-
-

15. Colon as file path separator

-
-MacOS can use the colon as a file path separator on its ADFS (file system). A typical ADFS file pathname on a hard disc might be:
-
-

-ADFS::4.$.Documents.Techwriter.Myfile
-

-
-I can't reproduce this on my (work) Mac, though, as it now uses the APFS file system. In essence, ADFS is an older file system, while APFS is a contemporary file system optimized for Apple's modern devices.
-
-https://social.jvns.ca/@b0rk/113041293527832730
-
-

16. Polyglots - programs written in multiple languages

-
-A coding polyglot is a program or script written so that it can be executed in multiple programming languages without modification. This is typically achieved by leveraging syntax overlaps or crafting valid and meaningful code in each targeted language. Polyglot programs are often created as a challenge or for demonstration purposes to showcase language similarities or clever coding techniques.
-
-Check out my very own polyglot:
-
-The fibonatti.pl.c Polyglot
-
-

17. Languages, where indices start at 1

-
-Array indices start at 1 instead of 0 in some programming languages, known as one-based indexing. This can be controversial because zero-based indexing is more common in popular languages like C, C++, Java, and Python. One-based indexing can lead to off-by-one errors when developers switch between languages with different indexing schemes.
-
-Languages with One-Based Indexing:
-
-

Fortran
MATLAB
Lua
R (for vectors and lists)
Smalltalk
Julia (by default, although zero-based indexing is also possible)

-foo.lua example:
-
- -

arr = {10, 20, 30, 40, 50}
-print(arr[1]) -- Accessing the first element
-

-
- -

❯ lua foo.lua
-10
-

-
-One-based indexing is more natural for human-readable, mathematical, and theoretical contexts, where counting traditionally starts from one.
-
-

18. Perl Poetry

-
-Perl Poetry is a playful and creative practice within the programming community where Perl code is written as a poem. These poems are crafted to be syntactically valid Perl code and make sense as poetic text, often with whimsical or humorous intent. This showcases Perl's flexibility and expressiveness, as well as the creativity of its programmers.
-
-See this Poetry of my own; the Perl interpreter does not yield any syntax error parsing that. But also, the Peom doesn't do anything useful then executed:
-
- -

# (C) 2006 by Paul C. Buetow
-
-Christmas:{time;#!!!
-
-Children: do tell $wishes;
-
-Santa: for $each (@children) { 
-BEGIN { read $each, $their, wishes and study them; use Memoize#ing
-
-} use constant gift, 'wrapping'; 
-package Gifts; pack $each, gift and bless $each and goto deliver
-or do import if not local $available,!!! HO, HO, HO;
-
-redo Santa, pipe $gifts, to_childs;
-redo Santa and do return if last one, is, delivered; 
-
-deliver: gift and require diagnostics if our $gifts ,not break;
-do{ use NEXT; time; tied $gifts} if broken and dump the, broken, ones;
-The_children: sleep and wait for (each %gift) and try { to => untie $gifts };
-
-redo Santa, pipe $gifts, to_childs;
-redo Santa and do return if last one, is, delivered; 
-
-The_christmas_tree: formline s/ /childrens/, $gifts;
-alarm and warn if not exists $Christmas{ tree}, @t, $ENV{HOME};  
-write <111
- and send the
-EMail
-;wait and redo deliver until defined local $tree;
-
-redo Santa, pipe $gifts, to_childs;
-redo Santa and do return if last one, is, delivered ;}
-
-END {} our $mission and do sleep until next Christmas ;}
-
-__END__
-
-This is perl, v5.8.8 built for i386-freebsd-64int
-

-
-More Perl Poetry of mine
-
-

19. CSS3 is turing complete

-
-CSS3 is Turing complete because it can simulate a Turing machine using only CSS animations and styles without any JavaScript or external logic. This is achieved by using keyframe animations to change the styles of HTML elements in a way that encodes computation, performing calculations and state transitions.
-
-Is CSS turing complete?
-
-It is surprising because CSS is primarily a styling language intended for the presentation layer of web pages, not for computation or logic. Its capability to perform complex computations defies its typical use case and showcases the unintended computational power that can emerge from the creative use of seemingly straightforward technologies.
-
-Check out this 100% CSS implementation of the Conways Game of Life:
-
-
-
-CSS Conways Game of Life
-
-Conway's Game of Life is Turing complete because it can simulate a universal Turing machine, meaning it can perform any computation that a computer can, given the right initial conditions and sufficient time and space. Suppose a language can implement Conway's Game of Life. In that case, it demonstrates the language's ability to handle complex state transitions and computations. It has the necessary constructs (like iteration, conditionals, and data manipulation) to simulate any algorithm, thus confirming its Turing completeness.
-
-

20. The biggest shell programs

-
-One would think that shell scripts are only suitable for small tasks. Well, I must be wrong, as there are huge shell programs out there (up to 87k LOC) which aren't auto-generated but hand-written!
-
-The Biggest Sell Programs in the World
-
-My Gemtexter (bash) is only 1329 LOC as of now. So it's tiny.
-
-Gemtexter - One Bash script to rule it all
-
-I hope you had some fun. E-Mail your comments to paul@nospam.buetow.org :-)
-
-Back to the main site
-

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts

2025-01-30T09:22:06+02:00

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts

-
-Published at 2025-01-30T09:22:06+02:00
-
-This is the third blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution we will use on FreeBSD-based physical machines.
-
-2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
-2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
-2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts (You are currently reading this)
-
-
-
-

-
-

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
⇢ Introduction
⇢ Changes since last time
⇢ ⇢ FreeBSD upgrade from 14.1 to 14.2
⇢ ⇢ A new home (behind the TV)
⇢ The UPS hardware
⇢ Configuring FreeBSD to Work with the UPS
⇢ ⇢ USB Device Detection
⇢ ⇢ apcupsd Installation
⇢ ⇢ UPS Connectivity Test
⇢ APC Info on Partner Nodes:
⇢ ⇢ Installation on partners
⇢ Power outage simulation
⇢ ⇢ Pulling the plug
⇢ ⇢ Restoring power

Introduction

-
-In this blog post, we are setting up the UPS for the cluster. A UPS, or Uninterruptible Power Supply, safeguards my cluster from unexpected power outages and surges. It acts as a backup battery that kicks in when the electricity cuts out—especially useful in my area, where power cuts are frequent—allowing for a graceful system shutdown and preventing data loss and corruption. This is especially important since I will also store some of my data on the f3s nodes.
-
-

Changes since last time

-
-

FreeBSD upgrade from 14.1 to 14.2

-
-There has been a new release since the last blog post in this series. The upgrade from 14.1 was as easy as:
-
- -

paul@f0: ~ % doas freebsd-update fetch
-paul@f0: ~ % doas freebsd-update install
-paul@f0: ~ % doas freebsd-update -r 14.2-RELEASE upgrade
-paul@f0: ~ % doas freebsd-update install
-paul@f0: ~ % doas shutdown -r now
-

-
-And after rebooting, I ran:
-
- -

paul@f0: ~ % doas freebsd-update install
-paul@f0: ~ % doas pkg update
-paul@f0: ~ % doas pkg upgrade
-paul@f0: ~ % doas shutdown -r now
-

-
-And after another reboot, I was on 14.2:
-
- -

paul@f0:~ % uname -a
-FreeBSD f0.lan.buetow.org 14.2-RELEASE FreeBSD 14.2-RELEASE 
- releng/14.2-n269506-c8918d6c7412 GENERIC amd64
-

-
-And, of course, I ran this on all 3 nodes!
-
-

A new home (behind the TV)

-
-I've put all the infrastructure behind my TV, as plenty of space is available. The TV hides most of the setup, which drastically improved the SAF (spouse acceptance factor).
-
-
-
-I got rid of the mini-switch I mentioned in the previous blog post. I have the TP-Link EAP615-Wall mounted on the wall nearby, which is my OpenWrt-powered Wi-Fi hotspot. It also has 3 Ethernet ports, to which I connected the Beelink nodes. That's the device you see at the very top.
-
-The Ethernet cables go downward through the cable boxes to the Beelink nodes. In addition to the Beelink f3s nodes, I connected the TP-Link to the UPS as well (not discussed further in this blog post, but the positive side effect is that my Wi-Fi will still work during a power loss for some time—and during a power cut, the Beelink nodes will still be able to communicate with each other).
-
-On the very left (the black box) is the UPS, with four power outlets. Three go to the Beelink nodes, and one goes to the TP-Link. A USB output is also connected to the first Beelink node, f0.
-
-On the very right (halfway hidden behind the TV) are the 3 Beelink nodes stacked on top of each other. The only downside (or upside?) is that my 14-month-old daughter is now chaos-testing the Beelink nodes, as the red power buttons (now reachable for her) are very attractive for her to press when passing by randomly. :-) Luckily, that will only cause graceful system shutdowns!
-
-

The UPS hardware

-
-I wanted a UPS that I could connect to via FreeBSD, and that would provide enough backup power to operate the cluster for a couple of minutes (it turned out to be around an hour, but this time will likely be shortened after future hardware upgrades, like additional drives and a backup enclosure) and to automatically initiate the shutdown of all the f3s nodes.
-
-I decided on the APC Back-UPS BX750MI model because:
-
-

Zero noise level when there is no power cut (some light noise when the battery is in operation during a power cut).
Cost: It is relatively affordable (not costing thousands).
USB connectivity: Can be connected via USB to one of the FreeBSD hosts to read the UPS status.
A power output of 750VA (or 410 watts), suitable for an hour of runtime for my f3s nodes (plus the Wi-Fi router).
Multiple power outlets: Can connect all 3 f3s nodes directly.
User-replaceable batteries: I can replace the batteries myself after two years or more (depending on usage).
Its compact design. Overall, I like how it looks.

-
-
-

Configuring FreeBSD to Work with the UPS

-
-

USB Device Detection

-
-Once plugged in via USB on FreeBSD, I could see the following in the kernel messages:
-
- -

paul@f0: ~ % doas dmesg | grep UPS
-ugen0.2:  at usbus0
-

-
-

apcupsd Installation

-
-To make use of the USB connection, the apcupsd package had to be installed:
-
- -

paul@f0: ~ % doas install apcupsd
-

-
-I have made the following modifications to the configuration file so that the UPS can be used via the USB interface:
-
- -

paul@f0:/usr/local/etc/apcupsd % diff -u apcupsd.conf.sample  apcupsd.conf
---- apcupsd.conf.sample 2024-11-01 16:40:42.000000000 +0200
-+++ apcupsd.conf        2024-12-03 10:58:24.009501000 +0200
-@@ -31,7 +31,7 @@
- #     940-1524C, 940-0024G, 940-0095A, 940-0095B,
- #     940-0095C, 940-0625A, M-04-02-2000
- #
--UPSCABLE smart
-+UPSCABLE usb
-
- # To get apcupsd to work, in addition to defining the cable
- # above, you must also define a UPSTYPE, which corresponds to
-@@ -88,8 +88,10 @@
- #                            that apcupsd binds to that particular unit
- #                            (helpful if you have more than one USB UPS).
- #
--UPSTYPE apcsmart
--DEVICE /dev/usv
-+UPSTYPE usb
-+DEVICE
-
- # POLLTIME 
- #   Interval (in seconds) at which apcupsd polls the UPS for status. This
-

-
-I left the remaining settings as the default ones; for example, the following are of main interest:
-
-

-# If during a power failure, the remaining battery percentage
-# (as reported by the UPS) is below or equal to BATTERYLEVEL,
-# apcupsd will initiate a system shutdown.
-BATTERYLEVEL 5
-
-# If during a power failure, the remaining runtime in minutes
-# (as calculated internally by the UPS) is below or equal to MINUTES,
-# apcupsd, will initiate a system shutdown.
-MINUTES 3
-

-
-I then enabled and started the daemon:
-
- -

paul@f0:/usr/local/etc/apcupsd % doas sysrc apcupsd_enable=YES
-apcupsd_enable:  -> YES
-paul@f0:/usr/local/etc/apcupsd % doas service apcupsd start
-Starting apcupsd.
-

-
-

UPS Connectivity Test

-
-And voila, I could now access the UPS information via the apcaccess command; how convenient :-) (I also read through the manual page, which provides a good understanding of what else can be done with it!).
-
- -

paul@f0:~ % apcaccess
-APC      : 001,035,0857
-DATE     : 2025-01-26 14:43:27 +0200
-HOSTNAME : f0.lan.buetow.org
-VERSION  : 3.14.14 (31 May 2016) freebsd
-UPSNAME  : f0.lan.buetow.org
-CABLE    : USB Cable
-DRIVER   : USB UPS Driver
-UPSMODE  : Stand Alone
-STARTTIME: 2025-01-26 14:43:25 +0200
-MODEL    : Back-UPS BX750MI
-STATUS   : ONLINE
-LINEV    : 230.0 Volts
-LOADPCT  : 4.0 Percent
-BCHARGE  : 100.0 Percent
-TIMELEFT : 65.3 Minutes
-MBATTCHG : 5 Percent
-MINTIMEL : 3 Minutes
-MAXTIME  : 0 Seconds
-SENSE    : Medium
-LOTRANS  : 145.0 Volts
-HITRANS  : 295.0 Volts
-ALARMDEL : No alarm
-BATTV    : 13.6 Volts
-LASTXFER : Automatic or explicit self test
-NUMXFERS : 0
-TONBATT  : 0 Seconds
-CUMONBATT: 0 Seconds
-XOFFBATT : N/A
-SELFTEST : NG
-STATFLAG : 0x05000008
-SERIALNO : 9B2414A03599
-BATTDATE : 2001-01-01
-NOMINV   : 230 Volts
-NOMBATTV : 12.0 Volts
-NOMPOWER : 410 Watts
-END APC  : 2025-01-26 14:44:06 +0200
-

-
-

APC Info on Partner Nodes:

-
-So far, so good. Host f0 would shut down itself when short on power. But what about the f1 and f2 nodes? They aren't connected directly to the UPS and, therefore, wouldn't know that their power is about to be cut off. For this, apcupsd running on the f1 and f2 nodes can be configured to retrieve UPS information via the network from the apcupsd server running on the f0 node, which is connected directly to the APC via USB.
-
-Of course, this won't work when f0 is down. In this case, no operational node would be connected to the UPS via USB; therefore, the current power status would not be known. However, I consider this a rare circumstance. Furthermore, in case of an f0 system crash, sudden power outages on the two other nodes would occur at different times making real data loss (the main concern here) less likely.
-
-And if f0 is down and f1 and f2 receive new data and crash midway, it's likely that a client (e.g., an Android app or another laptop) still has the data stored on it, making data recoverable and data loss overall nearly impossible. I'd receive an alert if any of the nodes go down (more on monitoring later in this blog series).
-
-

Installation on partners

-
-To do this, I installed apcupsd via doas pkg install apcupsd on f1 and f2, and then I could connect to it this way:
-
- -

paul@f1:~ % apcaccess -h f0.lan.buetow.org | grep Percent
-LOADPCT  : 12.0 Percent
-BCHARGE  : 94.0 Percent
-MBATTCHG : 5 Percent
-

-
-But I want the daemon to be configured and enabled in such a way that it connects to the master UPS node (the one with the UPS connected via USB) so that it can also initiate a system shutdown when the UPS battery reaches low levels. For that, apcupsd itself needs to be aware of the UPS status.
-
-On f1 and f2, I changed the configuration to use f0 (where apcupsd is listening) as a remote device. I also changed the MINUTES setting from 3 to 6 and the BATTERYLEVEL setting from 5 to 10 to ensure that the f1 and f2 nodes could still connect to the f0 node for UPS information before f0 decides to shut down itself. So f1 and f2 must shut down earlier than f0:
-
- -

paul@f2:/usr/local/etc/apcupsd % diff -u apcupsd.conf.sample apcupsd.conf
---- apcupsd.conf.sample 2024-11-01 16:40:42.000000000 +0200
-+++ apcupsd.conf        2025-01-26 15:52:45.108469000 +0200
-@@ -31,7 +31,7 @@
- #     940-1524C, 940-0024G, 940-0095A, 940-0095B,
- #     940-0095C, 940-0625A, M-04-02-2000
- #
--UPSCABLE smart
-+UPSCABLE ether
-
- # To get apcupsd to work, in addition to defining the cable
- # above, you must also define a UPSTYPE, which corresponds to
-@@ -52,7 +52,6 @@
- #                            Network Information Server. This is used if the
- #                            UPS powering your computer is connected to a
- #                            different computer for monitoring.
--#
- # snmp      hostname:port:vendor:community
- #                            SNMP network link to an SNMP-enabled UPS device.
- #                            Hostname is the ip address or hostname of the UPS
-@@ -88,8 +87,8 @@
- #                            that apcupsd binds to that particular unit
- #                            (helpful if you have more than one USB UPS).
- #
--UPSTYPE apcsmart
--DEVICE /dev/usv
-+UPSTYPE net
-+DEVICE f0.lan.buetow.org:3551
-
- # POLLTIME 
- #   Interval (in seconds) at which apcupsd polls the UPS for status. This
-@@ -147,12 +146,12 @@
- # If during a power failure, the remaining battery percentage
- # (as reported by the UPS) is below or equal to BATTERYLEVEL,
- # apcupsd will initiate a system shutdown.
--BATTERYLEVEL 5
-+BATTERYLEVEL 10
-
- # If during a power failure, the remaining runtime in minutes
- # (as calculated internally by the UPS) is below or equal to MINUTES,
- # apcupsd, will initiate a system shutdown.
--MINUTES 3
-+MINUTES 6
-
- # If during a power failure, the UPS has run on batteries for TIMEOUT
- # many seconds or longer, apcupsd will initiate a system shutdown.
-
-

-So I also ran the following commands on f1 and f2:
-
- -

paul@f1:/usr/local/etc/apcupsd % doas sysrc apcupsd_enable=YES
-apcupsd_enable:  -> YES
-paul@f1:/usr/local/etc/apcupsd % doas service apcupsd start
-Starting apcupsd.
-

-
-And then I was able to connect to localhost via the apcaccess command:
-
- -

paul@f1:~ % doas apcaccess | grep Percent
-LOADPCT  : 5.0 Percent
-BCHARGE  : 95.0 Percent
-MBATTCHG : 5 Percent
-

-
-

Power outage simulation

-
-

Pulling the plug

-
-I simulated a power outage by removing the power input from the APC. Immediately, the following message appeared on all the nodes:
-
-

-Broadcast Message from root@f0.lan.buetow.org
-        (no tty) at 15:03 EET...
-
-Power failure. Running on UPS batteries.                                              
-

-
-I ran the following command to confirm the available battery time:
-
- -

paul@f0:/usr/local/etc/apcupsd % apcaccess -p TIMELEFT
-63.9 Minutes
-

-
-And after around one hour (f1 and f2 a bit earlier, f0 a bit later due to the different BATTERYLEVEL and MINUTES settings outlined earlier), the following broadcast was sent out:
-
-

-Broadcast Message from root@f0.lan.buetow.org
-        (no tty) at 15:08 EET...
-
-        *** FINAL System shutdown message from root@f0.lan.buetow.org ***
-
-System going down IMMEDIATELY
-
-apcupsd initiated shutdown
-

-
-And all the nodes shut down safely before the UPS ran out of battery!
-
-

Restoring power

-
-After restoring power, I checked the logs in /var/log/daemon.log and found the following on all 3 nodes:
-
-

-Jan 26 17:36:24 f2 apcupsd[2159]: Power failure.
-Jan 26 17:36:30 f2 apcupsd[2159]: Running on UPS batteries.
-Jan 26 17:36:30 f2 apcupsd[2159]: Battery charge below low limit.
-Jan 26 17:36:30 f2 apcupsd[2159]: Initiating system shutdown!
-Jan 26 17:36:30 f2 apcupsd[2159]: User logins prohibited
-Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd exiting, signal 15
-Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded
-

-
-All good :-) See you in the next post of this series!
-
-Other BSD related posts are:
-
-2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts (You are currently reading this)
-2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
-2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
-2024-04-01 KISS high-availability with OpenBSD
-2024-01-13 One reason why I love OpenBSD
-2022-10-30 Installing DTail on OpenBSD
-2022-07-30 Let's Encrypt with OpenBSD and Rex
-2016-04-09 Jails and ZFS with Puppet on FreeBSD
-
-E-Mail your comments to paul@nospam.buetow.org :-)
-
-Back to the main site
-

Working with an SRE Interview

2025-01-15T00:16:04+02:00

Working with an SRE Interview

-
-Published at 2025-01-15T00:16:04+02:00
-
-I have been interviewed by Florian Buetow on cracking-ai-engineering.com about what it's like working with a Site Reliability Engineer from the point of view of a Software Engineer, Data Scientist, and AI Engineer.
-
-See original interview here
-Cracking AI Engineering
-
-Below, I am posting the interview here on my blog as well.
-
-

-
-

Working with an SRE Interview
⇢ Preamble
⇢ Introducing Paul
⇢ How did you get started?
⇢ Roles and Career Progression
⇢ Anecdotes and Best Practices
⇢ Working with Different Teams
⇢ Using AI Tools
⇢ SRE Learning Resources
⇢ Blogging
⇢ Wrap-up
⇢ Closing comments

Preamble

-
-In this insightful interview, Paul Bütow, a Principal Site Reliability Engineer at Mimecast, shares over a decade of experience in the field. Paul highlights the role of an Embedded SRE, emphasizing the importance of automation, observability, and effective incident management. We also focused on the key question of how you can work effectively with an SRE weather you are an individual contributor or a manager, a software engineer or data scientist. And how you can learn more about site reliability engineering.
-
-

Introducing Paul

-
-Hi Paul, please introduce yourself briefly to the audience. Who are you, what do you do for a living, and where do you work?
-
-My name is Paul Bütow, I work at Mimecast, and I’m a Principal Site Reliability Engineer there. I’ve been with Mimecast for almost ten years now. The company specializes in email security, including things like archiving, phishing detection, malware protection, and spam filtering.
-
-You mentioned that you’re an ‘Embedded SRE.’ What does that mean exactly?
-
-It means that I’m directly part of the software engineering team, not in a separate Ops department. I ensure that nothing is deployed manually, and everything runs through automation. I also set up monitoring and observability. These are two distinct aspects: monitoring alerts us when something breaks, while observability helps us identify trends. I also create runbooks so we know what to do when specific incidents occur frequently.
-
-Infrastructure SREs on the other hand handle the foundational setup, like providing the Kubernetes cluster itself or ensuring the operating systems are installed. They don't work on the application directly but ensure the base infrastructure is there for others to use. This works well when a company has multiple teams that need shared infrastructure.
-
-

How did you get started?

-
-How did your interest in Linux or FreeBSD start?
-
-It began during my school days. We had a PC with DOS at home, and I eventually bought Suse Linux 5.3. Shortly after, I discovered FreeBSD because I liked its handbook so much. I wanted to understand exactly how everything worked, so I also tried Linux from Scratch. That involves installing every package manually to gain a better understanding of operating systems.
-
-https://www.FreeBSD.org
-https://linuxfromscratch.org/
-
-And after school, you pursued computer science, correct?
-
-Exactly. I wasn’t sure at first whether I wanted to be a software developer or a system administrator. I applied for both and eventually accepted an offer as a Linux system administrator. This was before 'SRE' became a buzzword, but much of what I did back then-automation, infrastructure as code, monitoring-is now considered part of the typical SRE role.
-
-

Roles and Career Progression

-
-Tell us about how you joined Mimecast. When did you fully embrace the SRE role?
-
-I started as a Linux sysadmin at 1&1. I managed an ad server farm with hundreds of systems and later handled load balancers. Together with an architect, we managed F5 load balancers distributing around 2,000 services, including for portals like web.de and GMX. I also led the operations team technically for a while before moving to London to join Mimecast.
-
-At Mimecast, the job title was explicitly 'Site Reliability Engineer.' The biggest difference was that I was no longer in a separate Ops department but embedded directly within the storage and search backend team. I loved that because we could plan features together-from automation to measurability and observability. Mimecast also operates thousands of physical servers for email archiving, which was fascinating since I already had experience with large distributed systems at 1&1. It was the right step for me because it allowed me to work close to the code while remaining hands-on with infrastructure.
-
-What are the differences between SRE, DevOps, SysAdmin, and Architects?
-
-SREs are like the next step after SysAdmins. A SysAdmin might manually install servers, replace disks, or use simple scripts for automation, while SREs use infrastructure as code and focus on reliability through SLIs, SLOs, and automation. DevOps isn’t really a job-it’s more of a way of working, where developers are involved in operations tasks like setting up CI/CD pipelines or on-call shifts. Architects focus on designing systems and infrastructures, such as load balancers or distributed systems, working alongside SREs to ensure the systems meet the reliability and scalability requirements. The specific responsibilities of each role depend on the company, and there is often overlap.
-
-What are the most important reliability lessons you’ve learned so far?
-
-

Don’t leave SRE aspects as an afterthought. It’s much better to discuss automation, monitoring, SLIs, and SLOs early on. Traditional sysadmins often installed systems manually, but today, we do everything via infrastructure as code-using tools like Terraform or Puppet.
I also distinguish between monitoring and observability. Monitoring tells us, 'The server is down, alarm!' Observability dives deeper, showing trends like increasing latency so we can act proactively.
SLI, SLO, and SLA are core elements. We focus on what users actually experience-for example, how quickly an email is sent-and set our goals accordingly.
Runbooks are also crucial. When something goes wrong at night, you don’t want to start from scratch. A runbook outlines how to debug and resolve specific problems, saving time and reducing downtime.

Anecdotes and Best Practices

-
-Runbooks sound very practical. Can you explain how they’re used day-to-day?
-
-Runbooks are essentially guides for handling specific incidents. For instance, if a service won’t start, the runbook will specify where the logs are and which commands to use. Observability takes it a step further, helping us spot changes early-like rising error rates or latency-so we can address issues before they escalate.
-
-When should you decide to put something into a runbook, and when is it unnecessary?
-
-If an issue happens frequently, it should be documented in a runbook so that anyone, even someone new, can follow the steps to fix it. The idea is that 90% of the common incidents should be covered. For example, if a service is down, the runbook would specify where to find logs, which commands to check, and what actions to take. On the other hand, rare or complex issues, where the resolution depends heavily on context or varies each time, don’t make sense to include in detail. For those, it’s better to focus on general troubleshooting steps.
-
-How do you search for and find the correct runbooks?
-
-Runbooks should be linked directly in the alert you receive. For example, if you get an alert about a service not running, the alert will have a link to the runbook that tells you what to check, like logs or commands to run. Runbooks are best stored in an internal wiki, so if you don’t find the link in the alert, you know where to search. The important thing is that runbooks are easy to find and up to date because that’s what makes them useful during incidents.
-
-Do you have an interesting war story you can share with us?
-
-Sure. At 1&1, we had a proprietary ad server software that ran a SQL query during startup. The query got slower over time, eventually timing out and preventing the server from starting. Since we couldn’t access the source code, we searched the binary for the SQL and patched it. By pinpointing the issue, a developer was able to adjust the SQL. This collaboration between sysadmin and developer perspectives highlights the value of SRE work.
-
-

Working with Different Teams

-
-You’re embedded in a team-how does collaboration with developers work practically?
-
-We plan everything together from the start. If there’s a new feature, we discuss infrastructure, automated deployments, and monitoring right away. Developers are experts in the code, and I bring the infrastructure expertise. This avoids unpleasant surprises before going live.
-
-How about working with data scientists or ML engineers? Are there differences?
-
-The principles are the same. ML models also need to be deployed and monitored. You deal with monitoring, resource allocation, and identifying performance drops. Whether it’s a microservice or an ML job, at the end of the day, it’s all running on servers or clusters that must remain stable.
-
-What about working with managers or the FinOps team?
-
-We often discuss costs, especially in the cloud, where scaling up resources is easy. It’s crucial to know our metrics: do we have enough capacity? Do we need all instances? Or is the CPU only at 5% utilization? This data helps managers decide whether the budget is sufficient or if optimizations are needed.
-
-Do you have practical tips for working with SREs?
-
-Yes, I have a few:
-
-

Early involvement: Include SREs from the beginning in your project.
Runbooks & documentation: Document recurring errors.
Try first: Try to understand the issue yourself before immediately asking the SRE.
Basic infra knowledge: Kubernetes and Terraform aren’t magic. Some basic understanding helps every developer.

Using AI Tools

-
-Let’s talk about AI. How do you use it in your daily work?
-
-For boilerplate code, like Terraform snippets, I often use ChatGPT. It saves time, although I always review and adjust the output. Log analysis is another exciting application. Instead of manually going through millions of lines, AI can summarize key outliers or errors.
-
-Do you think AI could largely replace SREs or significantly change the role?
-
-I see AI as an additional tool. SRE requires a deep understanding of how distributed systems work internally. While AI can assist with routine tasks or quickly detect anomalies, human expertise is indispensable for complex issues.
-
-

SRE Learning Resources

-
-What resources would you recommend for learning about SRE?
-
-The Google SRE book is a classic, though a bit dry. I really like 'Seeking SRE,' as it offers various perspectives on SRE, with many practical stories from different companies.
-
-https://sre.google/books/
-Seeking SRE
-
-Do you have a podcast recommendation?
-
-The Google SRE prodcast is quite interesting. It offers insights into how Google approaches SRE, along with perspectives from external guests.
-
-https://sre.google/prodcast/
-
-

Blogging

-
-You also have a blog. What motivates you to write regularly?
-
-Writing helps me learn the most. It also serves as a personal reference. Sometimes I look up how I solved a problem a year ago. And of course, others tackling similar projects might find inspiration in my posts.
-
-What do you blog about?
-
-Mostly technical topics I find exciting, like homelab projects, Kubernetes, or book summaries on IT and productivity. It’s a personal blog, so I write about what I enjoy.
-
-

Wrap-up

-
-To wrap up, what are three things every team should keep in mind for stability?
-
-First, maintain runbooks and documentation to avoid chaos at night. Second, automate everything-manual installs in production are risky. Third, define SLIs, SLOs, and SLAs early so everyone knows what we’re monitoring and guaranteeing.
-
-Is there a motto or mindset that particularly inspires you as an SRE?
-
-"Keep it simple and stupid"-KISS. Not everything has to be overly complex. And always stay curious. I’m still fascinated by how systems work under the hood.
-
-Where can people find you online?
-
-You can find links to my socials on my website paul.buetow.org
-I regularly post articles and link to everything else I’m working on outside of work.
-
-https://paul.buetow.org
-
-Thank you very much for your time and this insightful interview into the world of site reliability engineering
-
-My pleasure, this was fun.
-
-

Closing comments

-
-Dear reader, I hope this conversation with Paul Bütow provided an exciting peak into the world of Site Reliability Engineering. Whether you’re a software developer, data scientist, ML engineer, or manager, reliable systems are always a team effort. Hopefully, you’ve taken some insights or tips from Paul’s experiences for your own team or next project. Thanks for joining us, and best of luck refining your own SRE practices!
-
-E-Mail your comments to paul@nospam.buetow.org or contact Florian via the Cracking AI Engineering :-)
-
-Back to the main site
-

Posts from October to December 2024

2024-12-31T18:09:58+02:00

Posts from October to December 2024

-
-Published at 2024-12-31T18:09:58+02:00
-
-Happy new year!
-
-These are my social media posts from the last three months. I keep them here to reflect on them and also to not lose them. Social media networks come and go and are not under my control, but my domain is here to stay.
-
-These are from Mastodon and LinkedIn. Have a look at my about page for my social media profiles. This list is generated with Gos, my social media platform sharing tool.
-
-My about page
-https://codeberg.org/snonux/gos
-
-

-
-

Posts from October to December 2024
Posts for 202410 202411 202412
⇢ October 2024
⇢ ⇢ First on-call experience in a startup. Doesn't ...
⇢ ⇢ Reviewing your own PR or MR before asking ...
⇢ ⇢ Fun with defer in #golang, I did't know, that ...
⇢ ⇢ I have been in incidents. Understandably, ...
⇢ ⇢ Little tips using strings in #golang and I ...
⇢ ⇢ Reading this post about #rust (especially the ...
⇢ ⇢ The opposite of #ChaosMonkey ... ...
⇢ November 2024
⇢ ⇢ I just became a Silver Patreon for OSnews. What ...
⇢ ⇢ Until now, I wasn't aware, that Go is under a ...
⇢ ⇢ These are some book notes from "Staff Engineer" ...
⇢ ⇢ Looking at #Kubernetes, it's pretty much ...
⇢ ⇢ There has been an outage at the upstream ...
⇢ ⇢ One of the more confusing parts in Go, nil ...
⇢ ⇢ Agreeably, writing down with Diagrams helps you ...
⇢ ⇢ I like the idea of types in Ruby. Raku is ...
⇢ ⇢ So, #Haskell is better suited for general ...
⇢ ⇢ At first, functional options add a bit of ...
⇢ ⇢ Revamping my home lab a little bit. #freebsd ...
⇢ ⇢ Wondering to which #web #browser I should ...
⇢ ⇢ eks-node-viewer is a nifty tool, showing the ...
⇢ ⇢ Have put more Photos on - On my static photo ...
⇢ ⇢ In Go, passing pointers are not automatically ...
⇢ ⇢ Myself being part of an on-call rotations over ...
⇢ ⇢ Feels good to code in my old love #Perl again ...
⇢ ⇢ This is an interactive summary of the Go ...
⇢ December 2024
⇢ ⇢ Thats unexpected, you cant remove a NaN key ...
⇢ ⇢ My second blog post about revamping my home lab ...
⇢ ⇢ Very insightful article about tech hiring in ...
⇢ ⇢ for #bpf #ebpf performance debugging, have ...
⇢ ⇢ 89 things he/she knows about Git commits is a ...
⇢ ⇢ I found that working on multiple side projects ...
⇢ ⇢ Agreed? Agreed. Besides #Ruby, I would also ...
⇢ ⇢ Plan9 assembly format in Go, but wait, it's not ...
⇢ ⇢ This is a neat blog post about the Helix text ...
⇢ ⇢ This blog post is basically a rant against ...
⇢ ⇢ Quick trick to get Helix themes selected ...
⇢ ⇢ Example where complexity attacks you from ...
⇢ ⇢ LLMs for Ops? Summaries of logs, probabilities ...
⇢ ⇢ Excellent article about your dream Product ...
⇢ ⇢ I just finished reading all chapters of CPU ...
⇢ ⇢ Indeed, useful to know this stuff! #sre ...
⇢ ⇢ It's the small things, which make Unix like ...
⇢ ⇢ My New Year's resolution is not to start any ...

Posts for 202410 202411 202412

-
-

October 2024

-
-

First on-call experience in a startup. Doesn't ...

-
-First on-call experience in a startup. Doesn't sound a lot of fun! But the lessons were learned! #sre
-
-ntietz.com/blog/lessons-from-my-first-on-call/
-
-

Reviewing your own PR or MR before asking ...

-
-Reviewing your own PR or MR before asking others to review it makes a lot of sense. Have seen so many silly mistakes which would have been avoided. Saving time for the real reviewer.
-
-www.jvt.me/posts/2019/01/12/self-code-review/
-
-

Fun with defer in #golang, I did't know, that ...

-
-Fun with defer in #golang, I did't know, that a defer object can either be heap or stack allocated. And there are some rules for inlining, too.
-
-victoriametrics.com/blog/defer-in-go/
-
-

I have been in incidents. Understandably, ...

-
-I have been in incidents. Understandably, everyone wants the issue to be resolved as quickly and others want to know how long TTR will be. IMHO, providing no estimates at all is no solution either. So maybe give a rough estimate but clearly communicate that the estimate is rough and that X, Y, and Z can interfere, meaning there is a chance it will take longer to resolve the incident. Just my thought. What's yours?
-
-firehydrant.com/blog/hot-take-dont-provide-incident-resolution-estimates/
-
-

Little tips using strings in #golang and I ...

-
-Little tips using strings in #golang and I personally think one must look more into the std lib (not just for strings, also for slices, maps,...), there are tons of useful helper functions.
-
-www.calhoun.io/6-tips-for-using-strings-in-go/
-
-

Reading this post about #rust (especially the ...

-
-Reading this post about #rust (especially the first part), I think I made a good choice in deciding to dive into #golang instead. There was a point where I wanted to learn a new programming language, and Rust was on my list of choices. I think the Go project does a much better job of deciding what goes into the language and how. What are your thoughts?
-
-josephg.com/blog/rewriting-rust/
-
-

The opposite of #ChaosMonkey ... ...

-
-The opposite of #ChaosMonkey ... automatically repairing and healing services helping to reduce manual toil work. Runbooks and scripts are only the first step, followed by a fully blown service written in Go. Could be useful, but IMHO why not rather address the root causes of the manual toil work? #sre
-
-blog.cloudflare.com/nl-nl/improving-platform-resilience-at-cloudflare/
-
-

November 2024

-
-

I just became a Silver Patreon for OSnews. What ...

-
-I just became a Silver Patreon for OSnews. What is OSnews? It is an independent news site about IT. It is slightly independent and, at times, alternative. I have enjoyed it since my early student days. This one and other projects I financially support are listed here:
-
-foo.zone/gemfeed/2024-09-07-projects-i-support.gmi (Gemini)
-foo.zone/gemfeed/2024-09-07-projects-i-support.html
-
-

Until now, I wasn't aware, that Go is under a ...

-
-Until now, I wasn't aware, that Go is under a BSD-style license (3-clause as it seems). Neat. I don't know why, but I always was under the impression it would be MIT. #bsd #golang
-
-go.dev/LICENSE
-
-

These are some book notes from "Staff Engineer" ...

-
-These are some book notes from "Staff Engineer" – there is some really good insight into what is expected from a Staff Engineer and beyond in the industry. I wish I had read the book earlier.
-
-foo.zone/gemfeed/2024-10-24-staff-engineer-book-notes.gmi (Gemini)
-foo.zone/gemfeed/2024-10-24-staff-engineer-book-notes.html
-
-

Looking at #Kubernetes, it's pretty much ...

-
-Looking at #Kubernetes, it's pretty much following the Unix way of doing things. It has many tools, but each tool has its own single purpose: DNS, scheduling, container runtime, various controllers, networking, observability, alerting, and more services in the control plane. Everything is managed by different services or plugins, mostly running in their dedicated pods. They don't communicate through pipes, but network sockets, though. #k8s
-
-

There has been an outage at the upstream ...

-
-There has been an outage at the upstream network provider for OpenBSD.Amsterdam (hoster, I am using). This was the first real-world test for my KISS HA setup, and it worked flawlessly! All my sites and services failed over automatically to my other #OpenBSD VM!
-
-foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi (Gemini)
-foo.zone/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html
-openbsd.amsterdam/
-
-

One of the more confusing parts in Go, nil ...

-
-One of the more confusing parts in Go, nil values vs nil errors: #golang
-
-unexpected-go.com/nil-errors-that-are-non-nil-errors.html
-
-

Agreeably, writing down with Diagrams helps you ...

-
-Agreeably, writing down with Diagrams helps you to think things more through. And keeps others on the same page. Only worth for projects from a certain size, IMHO.
-
-ntietz.com/blog/reasons-to-write-design-docs/
-
-

I like the idea of types in Ruby. Raku is ...

-
-I like the idea of types in Ruby. Raku is supports that already, but in Ruby, you must specify the types in a separate .rbs file, which is, in my opinion, cumbersome and is a reason not to use it extensively for now. I believe there are efforts to embed the type information in the standard .rb files, and that the .rbs is just an experiment to see how types could work out without introducing changes into the core Ruby language itself right now? #Ruby #RakuLang
-
-github.com/ruby/rbs
-
-

So, #Haskell is better suited for general ...

-
-So, #Haskell is better suited for general purpose than #Rust? I thought deploying something in Haskell means publishing an academic paper :-) Interesting rant about Rust, though:
-
-chrisdone.com/posts/rust/
-
-

At first, functional options add a bit of ...

-
-At first, functional options add a bit of boilerplate, but they turn out to be quite neat, especially when you have very long parameter lists that need to be made neat and tidy. #golang
-
-www.calhoun.io/using-functional-options-instead-of-method-chaining-in-go/
-
-

Revamping my home lab a little bit. #freebsd ...

-
-Revamping my home lab a little bit. #freebsd #bhyve #rocky #linux #vm #k3s #kubernetes #wireguard #zfs #nfs #ha #relayd #k8s #selfhosting #homelab
-
-foo.zone/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi (Gemini)
-foo.zone/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.html
-
-

Wondering to which #web #browser I should ...

-
-Wondering to which #web #browser I should switch now personally ...
-
-www.osnews.com/story/141100/mozilla-fo..-..dvocacy-for-open-web-privacy-and-more/
-
-

eks-node-viewer is a nifty tool, showing the ...

-
-eks-node-viewer is a nifty tool, showing the compute nodes currently in use in the #EKS cluster. especially useful when dynamically allocating nodes with #karpenter or auto scaling groups.
-
-github.com/awslabs/eks-node-viewer
-
-

Have put more Photos on - On my static photo ...

-
-Have put more Photos on - On my static photo sites - Generated with a #bash script
-
-irregular.ninja
-
-

In Go, passing pointers are not automatically ...

-
-In Go, passing pointers are not automatically faster than values. Pointers often force the memory to be allocated on the heap, adding GC overhad. With values, Go can determine whether to put the memory on the stack instead. But with large structs/objects (how you want to call them) or if you want to modify state, then pointers are the semantic to use. #golang
-
-blog.boot.dev/golang/pointers-faster-than-values/
-
-

Myself being part of an on-call rotations over ...

-
-Myself being part of an on-call rotations over my whole professional life, just have learned this lesson "Tell people who are new to on-call: Just have fun" :-) This is a neat blog post to read:
-
-ntietz.com/blog/what-i-tell-people-new-to-oncall/
-
-

Feels good to code in my old love #Perl again ...

-
-Feels good to code in my old love #Perl again after a while. I am implementing a log parser for generating site stats of my personal homepage! :-) @Perl
-
-

This is an interactive summary of the Go ...

-
-This is an interactive summary of the Go release, with a lot of examples utilising iterators in the slices and map packages. Love it! #golang
-
-antonz.org/go-1-23/
-
-

December 2024

-
-

Thats unexpected, you cant remove a NaN key ...

-
-Thats unexpected, you cant remove a NaN key from a map without clearing it! #golang
-
-unexpected-go.com/you-cant-remove-a-nan-key-from-a-map-without-clearing-it.html
-
-

My second blog post about revamping my home lab ...

-
-My second blog post about revamping my home lab a little bit just hit the net. #FreeBSD #ZFS #n100 #k8s #k3s #kubernetes
-
-foo.zone/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi (Gemini)
-foo.zone/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.html
-
-

Very insightful article about tech hiring in ...

-
-Very insightful article about tech hiring in the age of LLMs. As an interviewer, I have experienced some of the scrnarios already first hand...
-
-newsletter.pragmaticengineer.com/p/how-genai-changes-tech-hiring
-
-

for #bpf #ebpf performance debugging, have ...

-
-for #bpf #ebpf performance debugging, have a look at bpftop from Netflix. A neat tool showing you the estimated CPU time and other performance statistics for all the BPF programs currently loaded into the #linux kernel. Highly recommend!
-
-github.com/Netflix/bpftop
-
-

89 things he/she knows about Git commits is a ...

-
-89 things he/she knows about Git commits is a neat list of #Git wisdoms
-
-www.jvt.me/posts/2024/07/12/things-know-commits/
-
-

I found that working on multiple side projects ...

-
-I found that working on multiple side projects concurrently is better than concentrating on just one. This seems inefficient at first, but whenever you tend to lose motivation, you can temporarily switch to another one with full élan. However, remember to stop starting and start finishing. This doesn't mean you should be working on 10+ (and a growing list of) side projects concurrently! Select your projects and commit to finishing them before starting the next thing. For example, my current limit of concurrent side projects is around five.
-
-

Agreed? Agreed. Besides #Ruby, I would also ...

-
-Agreed? Agreed. Besides #Ruby, I would also add #RakuLang and #Perl @Perl to the list of languages that are great for shell scripts - "Making Easy Things Easy and Hard Things Possible"
-
-lucasoshiro.github.io/posts-en/2024-06-17-ruby-shellscript/
-
-

Plan9 assembly format in Go, but wait, it's not ...

-
-Plan9 assembly format in Go, but wait, it's not the Operating System Plan9! #golang #rabbithole
-
-www.osnews.com/story/140941/go-plan9-memo-speeding-up-calculations-450/
-
-

This is a neat blog post about the Helix text ...

-
-This is a neat blog post about the Helix text editor, to which I personally switched around a year ago (from NeoVim). I should blog about my experience as well. To summarize: I am using it together with the terminal multiplexer #tmux. It doesn't bother me that Helix is purely terminal-based and therefore everything has to be in the same font. #HelixEditor
-
-jonathan-frere.com/posts/helix/
-
-

This blog post is basically a rant against ...

-
-This blog post is basically a rant against DataDog... Personally, I don't have much experience with DataDog (actually, I have never used it), but one reason to work with logs at my day job (with over 2,000 physical server machines) and to be cost-effective is by using dtail! #dtail #logs #logmanagement
-
-crys.site/blog/2024/reinventint-the-weel/
-dtail.dev
-
-

Quick trick to get Helix themes selected ...

-
-Quick trick to get Helix themes selected randomly #HelixEditor
-
-foo.zone/gemfeed/2024-12-15-random-helix-themes.gmi (Gemini)
-foo.zone/gemfeed/2024-12-15-random-helix-themes.html
-
-

Example where complexity attacks you from ...

-
-Example where complexity attacks you from behind #k8s #kubernetes #OpenAI
-
-surfingcomplexity.blog/2024/12/14/quic..-..ecent-openai-public-incident-write-up/
-
-

LLMs for Ops? Summaries of logs, probabilities ...

-
-LLMs for Ops? Summaries of logs, probabilities about correctness, auto-generating Ansible, some uses cases are there. Wouldn't trust it fully, though.
-
-youtu.be/WodaffxVq-E?si=noY0egrfl5izCSQI
-
-

Excellent article about your dream Product ...

-
-Excellent article about your dream Product Manager: Why every software team needs a product manager to thrive via @wallabagapp
-
-testdouble.com/insights/why-product-ma..-..s-accelerate-improve-software-delivery
-
-

I just finished reading all chapters of CPU ...

-
-I just finished reading all chapters of CPU land: ... not claiming to remember every detail, but it is a great refresher how CPUs and operating systems actually work under the hood when you execute a program, which we tend to forget in our higher abstraction world. I liked the "story" and some of the jokes along the way! Size wise, it is pretty digestable (not talking about books, but only 7 web articles/chapters)! #cpu #linux #unix #kernel #macOS
-
-cpu.land/
-
-

Indeed, useful to know this stuff! #sre ...

-
-Indeed, useful to know this stuff! #sre
-
-biriukov.dev/docs/resolver-dual-stack-..-..resolvers-and-dual-stack-applications/
-
-

It's the small things, which make Unix like ...

-
-It's the small things, which make Unix like systems, like GNU/Linux, interesting. Didn't know about this #GNU #Tar behaviour yet:
-
-xeiaso.net/notes/2024/pop-quiz-tar/
-
-

My New Year's resolution is not to start any ...

-
-My New Year's resolution is not to start any new non-fiction books (or only very few) but to re-read and listen to my favorites, which I read to reflect on and see things from different perspectives. Every time you re-read a book, you gain new insights.17491
-
-Other related posts:
-
-2025-01-01 Posts from October to December 2024 (You are currently reading this)
-
-E-Mail your comments to paul@nospam.buetow.org :-)
-
-Back to the main site
-

foo.zone feed

Random Weird Things - Part Ⅱ

Random Weird Things - Part Ⅱ

Table of Contents

11. The SQLite codebase is a gem

Go Programming

12. Official Go font

13. Go functions can have methods

macOS

14. ß and ss are treated the same

15. Colon as file path separator

16. Polyglots - programs written in multiple languages

17. Languages, where indices start at 1

18. Perl Poetry

19. CSS3 is turing complete

20. The biggest shell programs

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts

f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts

Table of Contents

Introduction

Changes since last time

FreeBSD upgrade from 14.1 to 14.2

A new home (behind the TV)

The UPS hardware

Configuring FreeBSD to Work with the UPS

USB Device Detection

apcupsd Installation

UPS Connectivity Test

APC Info on Partner Nodes:

Installation on partners

Power outage simulation

Pulling the plug

Restoring power

Working with an SRE Interview

Working with an SRE Interview

Table of Contents

Preamble

Introducing Paul

How did you get started?

Roles and Career Progression

Anecdotes and Best Practices

Working with Different Teams

Using AI Tools

SRE Learning Resources

Blogging

Wrap-up

Closing comments

Posts from October to December 2024

Posts from October to December 2024

Table of Contents

Posts for 202410 202411 202412

October 2024

First on-call experience in a startup. Doesn't ...

Reviewing your own PR or MR before asking ...

Fun with defer in #golang, I did't know, that ...

I have been in incidents. Understandably, ...

Little tips using strings in #golang and I ...

Reading this post about #rust (especially the ...

The opposite of #ChaosMonkey ... ...

November 2024

I just became a Silver Patreon for OSnews. What ...

Until now, I wasn't aware, that Go is under a ...

These are some book notes from "Staff Engineer" ...

Looking at #Kubernetes, it's pretty much ...

There has been an outage at the upstream ...

One of the more confusing parts in Go, nil ...

Agreeably, writing down with Diagrams helps you ...

I like the idea of types in Ruby. Raku is ...

So, #Haskell is better suited for general ...

At first, functional options add a bit of ...

Revamping my home lab a little bit. #freebsd ...

Wondering to which #web #browser I should ...

eks-node-viewer is a nifty tool, showing the ...

Have put more Photos on - On my static photo ...

In Go, passing pointers are not automatically ...

Myself being part of an on-call rotations over ...

Feels good to code in my old love #Perl again ...

This is an interactive summary of the Go ...

December 2024

Thats unexpected, you cant remove a NaN key ...