From 157c9b2080a3f41eea0eeba11f6ef307f40e9b9e Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 2 Oct 2025 11:31:40 +0300 Subject: Update content for gemtext --- gemfeed/atom.xml | 1472 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 1104 insertions(+), 368 deletions(-) (limited to 'gemfeed/atom.xml') diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 7a058662..68a3a2cc 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,11 +1,1092 @@ - 2025-09-29T09:38:00+03:00 + 2025-10-02T11:30:14+03:00 foo.zone feed To be in the .zone! gemini://foo.zone/ + + f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments + + gemini://foo.zone/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi + 2025-10-02T11:27:19+03:00 + + Paul Buetow aka snonux + paul@dev.buetow.org + + This is the seventh blog post about the f3s series for my self-hosting demands in a home lab. f3s? The 'f' stands for FreeBSD, and the '3s' stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines. + +
+

f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments


+
+Published at 2025-10-02T11:27:19+03:00
+
+This is the seventh blog post about the f3s series for my self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.
+
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
+2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
+2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)
+
+f3s logo
+
+

Table of Contents


+
+
+

Introduction


+
+In this blog post, I am finally going to install k3s (the Kubernetes distribution I use) to the whole setup and deploy the first workloads (helm charts, and a private registry) to it.
+
+https://k3s.io
+
+

Updating


+
+Before proceeding, I bring all systems involved up-to-date. On all three Rocky Linux 9 boxes r0, r1, and r2:
+
+ +
dnf update -y
+reboot
+
+
+On the FreeBSD hosts, I upgraded from FreeBSD 14.2 to 14.3-RELEASE, running this on all three hosts f0, f1 and f2:
+
+ +
paul@f0:~ % doas freebsd-update fetch
+paul@f0:~ % doas freebsd-update install
+paul@f0:~ % doas reboot
+.
+.
+.
+paul@f0:~ % doas freebsd-update -r 14.3-RELEASE upgrade
+paul@f0:~ % doas freebsd-update install
+paul@f0:~ % doas freebsd-update install
+paul@f0:~ % doas reboot
+.
+.
+.
+paul@f0:~ % doas freebsd-update install
+paul@f0:~ % doas pkg update
+paul@f0:~ % doas pkg upgrade
+paul@f0:~ % doas reboot
+.
+.
+.
+paul@f0:~ % uname -a
+FreeBSD f0.lan.buetow.org 14.3-RELEASE FreeBSD 14.3-RELEASE
+        releng/14.3-n271432-8c9ce319fef7 GENERIC amd64
+
+
+

Installing k3s


+
+

Generating K3S_TOKEN and starting the first k3s node


+
+I generated the k3s token on my Fedora laptop with pwgen -n 32 and selected one of the results. Then, on all three r hosts, I ran the following (replace SECRET_TOKEN with the actual secret):
+
+ +
[root@r0 ~]# echo -n SECRET_TOKEN > ~/.k3s_token
+
+
+The following steps are also documented on the k3s website:
+
+https://docs.k3s.io/datastore/ha-embedded
+
+To bootstrap k3s on the first node, I ran this on r0:
+
+ +
[root@r0 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
+        sh -s - server --cluster-init --tls-san=r0.wg0.wan.buetow.org
+[INFO]  Finding release for channel stable
+[INFO]  Using v1.32.6+k3s1 as release
+.
+.
+.
+[INFO]  systemd: Starting k3s
+
+
+

Adding the remaining nodes to the cluster


+
+Then I ran on the other two nodes r1 and r2:
+
+ +
[root@r1 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
+        sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \
+        --tls-san=r1.wg0.wan.buetow.org
+
+[root@r2 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
+        sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \
+        --tls-san=r2.wg0.wan.buetow.org
+.
+.
+.
+
+
+
+Once done, I had a three-node Kubernetes cluster control plane:
+
+ +
[root@r0 ~]# kubectl get nodes
+NAME                STATUS   ROLES                       AGE     VERSION
+r0.lan.buetow.org   Ready    control-plane,etcd,master   4m44s   v1.32.6+k3s1
+r1.lan.buetow.org   Ready    control-plane,etcd,master   3m13s   v1.32.6+k3s1
+r2.lan.buetow.org   Ready    control-plane,etcd,master   30s     v1.32.6+k3s1
+
+[root@r0 ~]# kubectl get pods --all-namespaces
+NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
+kube-system   coredns-5688667fd4-fs2jj                  1/1     Running     0          5m27s
+kube-system   helm-install-traefik-crd-f9hgd            0/1     Completed   0          5m27s
+kube-system   helm-install-traefik-zqqqk                0/1     Completed   2          5m27s
+kube-system   local-path-provisioner-774c6665dc-jqlnc   1/1     Running     0          5m27s
+kube-system   metrics-server-6f4c6675d5-5xpmp           1/1     Running     0          5m27s
+kube-system   svclb-traefik-411cec5b-cdp2l              2/2     Running     0          78s
+kube-system   svclb-traefik-411cec5b-f625r              2/2     Running     0          4m58s
+kube-system   svclb-traefik-411cec5b-twrd7              2/2     Running     0          4m2s
+kube-system   traefik-c98fdf6fb-lt6fx                   1/1     Running     0          4m58s
+
+
+In order to connect with kubectl from my Fedora laptop, I had to copy /etc/rancher/k3s/k3s.yaml from r0 to ~/.kube/config and then replace the value of the server field with r0.lan.buetow.org. kubectl can now manage the cluster. Note that this step has to be repeated when I want to connect to another node of the cluster (e.g. when r0 is down).
+
+

Test deployments


+
+

Test deployment to Kubernetes


+
+Let's create a test namespace:
+
+ +
> ~ kubectl create namespace test
+namespace/test created
+
+> ~ kubectl get namespaces
+NAME              STATUS   AGE
+default           Active   6h11m
+kube-node-lease   Active   6h11m
+kube-public       Active   6h11m
+kube-system       Active   6h11m
+test              Active   5s
+
+> ~ kubectl config set-context --current --namespace=test
+Context "default" modified.
+
+
+And let's also create an Apache test pod:
+
+ +
> ~ cat <<END > apache-deployment.yaml
+# Apache HTTP Server Deployment
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: apache-deployment
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: apache
+  template:
+    metadata:
+      labels:
+        app: apache
+    spec:
+      containers:
+      - name: apache
+        image: httpd:latest
+        ports:
+        # Container port where Apache listens
+        - containerPort: 80
+END
+
+> ~ kubectl apply -f apache-deployment.yaml
+deployment.apps/apache-deployment created
+
+> ~ kubectl get all
+NAME                                     READY   STATUS    RESTARTS   AGE
+pod/apache-deployment-5fd955856f-4pjmf   1/1     Running   0          7s
+
+NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
+deployment.apps/apache-deployment   1/1     1            1           7s
+
+NAME                                           DESIRED   CURRENT   READY   AGE
+replicaset.apps/apache-deployment-5fd955856f   1         1         1       7s
+
+
+Let's also create a service:
+
+ +
> ~ cat <<END > apache-service.yaml
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: apache
+  name: apache-service
+spec:
+  ports:
+    - name: web
+      port: 80
+      protocol: TCP
+      # Expose port 80 on the service
+      targetPort: 80
+  selector:
+  # Link this service to pods with the label app=apache
+    app: apache
+END
+
+> ~ kubectl apply -f apache-service.yaml
+service/apache-service created
+
+> ~ kubectl get service
+NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
+apache-service   ClusterIP   10.43.249.165   <none>        80/TCP    4s
+
+
+Now let's create an ingress:
+
+Note: I've modified the hosts listed in this example after I published this blog post to ensure that there aren't any bots scraping it.
+
+ +
> ~ cat <<END > apache-ingress.yaml
+
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: apache-ingress
+  namespace: test
+  annotations:
+    spec.ingressClassName: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: web
+spec:
+  rules:
+    - host: f3s.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: apache-service
+                port:
+                  number: 80
+    - host: standby.f3s.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: apache-service
+                port:
+                  number: 80
+    - host: www.f3s.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: apache-service
+                port:
+                  number: 80
+END
+
+> ~ kubectl apply -f apache-ingress.yaml
+ingress.networking.k8s.io/apache-ingress created
+
+> ~ kubectl describe ingress
+Name:             apache-ingress
+Labels:           <none>
+Namespace:        test
+Address:          192.168.1.120,192.168.1.121,192.168.1.122
+Ingress Class:    traefik
+Default backend:  <default>
+Rules:
+  Host                    Path  Backends
+  ----                    ----  --------
+  f3s.foo.zone
+                          /   apache-service:80 (10.42.1.11:80)
+  standby.f3s.foo.zone
+                          /   apache-service:80 (10.42.1.11:80)
+  www.f3s.foo.zone
+                          /   apache-service:80 (10.42.1.11:80)
+Annotations:              spec.ingressClassName: traefik
+                          traefik.ingress.kubernetes.io/router.entrypoints: web
+Events:                   <none>
+
+
+Notes:
+
+
    +
  • In the ingress, I use plain HTTP (web) for the Traefik rule, as all the "production" traffic will be routed through a WireGuard tunnel anyway, as I will show later.
    • +

+So I tested the Apache web server through the ingress rule:
+
+ +
> ~ curl -H "Host: www.f3s.foo.zone" http://r0.lan.buetow.org:80
+<html><body><h1>It works!</h1></body></html>
+
+
+

Test deployment with persistent volume claim


+
+Next, I modified the Apache example to serve the htdocs directory from the NFS share I created in the previous blog post. I used the following manifests. Most of them are the same as before, except for the persistent volume claim and the volume mount in the Apache deployment.
+
+ +
> ~ cat <<END > apache-deployment.yaml
+# Apache HTTP Server Deployment
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: apache-deployment
+  namespace: test
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: apache
+  template:
+    metadata:
+      labels:
+        app: apache
+    spec:
+      containers:
+      - name: apache
+        image: httpd:latest
+        ports:
+        # Container port where Apache listens
+        - containerPort: 80
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 80
+          initialDelaySeconds: 5
+          periodSeconds: 10
+        livenessProbe:
+          httpGet:
+            path: /
+            port: 80
+          initialDelaySeconds: 15
+          periodSeconds: 10
+        volumeMounts:
+        - name: apache-htdocs
+          mountPath: /usr/local/apache2/htdocs/
+      volumes:
+      - name: apache-htdocs
+        persistentVolumeClaim:
+          claimName: example-apache-pvc
+END
+
+> ~ cat <<END > apache-ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: apache-ingress
+  namespace: test
+  annotations:
+    spec.ingressClassName: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: web
+spec:
+  rules:
+    - host: f3s.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: apache-service
+                port:
+                  number: 80
+    - host: standby.f3s.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: apache-service
+                port:
+                  number: 80
+    - host: www.f3s.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: apache-service
+                port:
+                  number: 80
+END
+
+> ~ cat <<END > apache-persistent-volume.yaml
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: example-apache-pv
+spec:
+  capacity:
+    storage: 1Gi
+  volumeMode: Filesystem
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /data/nfs/k3svolumes/example-apache-volume-claim
+    type: Directory
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: example-apache-pvc
+  namespace: test
+spec:
+  storageClassName: ""
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+END
+
+> ~ cat <<END > apache-service.yaml
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: apache
+  name: apache-service
+  namespace: test
+spec:
+  ports:
+    - name: web
+      port: 80
+      protocol: TCP
+      # Expose port 80 on the service
+      targetPort: 80
+  selector:
+  # Link this service to pods with the label app=apache
+    app: apache
+END
+
+
+I applied the manifests:
+
+ +
> ~ kubectl apply -f apache-persistent-volume.yaml
+> ~ kubectl apply -f apache-service.yaml
+> ~ kubectl apply -f apache-deployment.yaml
+> ~ kubectl apply -f apache-ingress.yaml
+
+
+Looking at the deployment, I could see it failed because the directory didn't exist yet on the NFS share (note that I also increased the replica count to 2 so if one node goes down there's already a replica running on another node for faster failover):
+
+ +
> ~ kubectl get pods
+NAME                                 READY   STATUS              RESTARTS   AGE
+apache-deployment-5b96bd6b6b-fv2jx   0/1     ContainerCreating   0          9m15s
+apache-deployment-5b96bd6b6b-ax2ji   0/1     ContainerCreating   0          9m15s
+
+> ~ kubectl describe pod apache-deployment-5b96bd6b6b-fv2jx | tail -n 5
+Events:
+  Type     Reason       Age                   From               Message
+  ----     ------       ----                  ----               -------
+  Normal   Scheduled    9m34s                 default-scheduler  Successfully
+    assigned test/apache-deployment-5b96bd6b6b-fv2jx to r2.lan.buetow.org
+  Warning  FailedMount  80s (x12 over 9m34s)  kubelet            MountVolume.SetUp
+    failed for volume "example-apache-pv" : hostPath type check failed:
+    /data/nfs/k3svolumes/example-apache is not a directory
+
+
+That's intentional—I needed to create the directory on the NFS share first, so I did that (e.g. on r0):
+
+ +
[root@r0 ~]# mkdir /data/nfs/k3svolumes/example-apache-volume-claim/
+
+[root@r0 ~]# cat <<END > /data/nfs/k3svolumes/example-apache-volume-claim/index.html
+<!DOCTYPE html>
+<html>
+<head>
+  <title>Hello, it works</title>
+</head>
+<body>
+  <h1>Hello, it works!</h1>
+  <p>This site is served via a PVC!</p>
+</body>
+</html>
+END
+
+
+The index.html file gives us some actual content to serve. After deleting the pod, it recreates itself and the volume mounts correctly:
+
+ +
> ~ kubectl delete pod apache-deployment-5b96bd6b6b-fv2jx
+
+> ~ curl -H "Host: www.f3s.foo.zone" http://r0.lan.buetow.org:80
+<!DOCTYPE html>
+<html>
+<head>
+  <title>Hello, it works</title>
+</head>
+<body>
+  <h1>Hello, it works!</h1>
+  <p>This site is served via a PVC!</p>
+</body>
+</html>
+
+
+

Scaling Traefik for faster failover


+
+Traefik (used for ingress on k3s) ships with a single replica by default, but for faster failover I bumped it to two replicas so each worker node runs one pod. That way, if a node disappears, the service stays up while Kubernetes schedules a replacement. Here's the command I used:
+
+ +
> ~ kubectl -n kube-system scale deployment traefik --replicas=2
+
+
+And the result:
+
+ +
> ~ kubectl -n kube-system get pods -l app.kubernetes.io/name=traefik
+kube-system   traefik-c98fdf6fb-97kqk   1/1   Running   19 (53d ago)   64d
+kube-system   traefik-c98fdf6fb-9npg2   1/1   Running   11 (53d ago)   61d
+
+
+

Make it accessible from the public internet


+
+Next, I made this accessible through the public internet via the www.f3s.foo.zone hosts. As a reminder from part 1 of this series, I reviewed the section titled "OpenBSD/relayd to the rescue for external connectivity":
+
+f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+
+All apps should be reachable through the internet (e.g., from my phone or computer when travelling). For external connectivity and TLS management, I've got two OpenBSD VMs (one hosted by OpenBSD Amsterdam and another hosted by Hetzner) handling public-facing services like DNS, relaying traffic, and automating Let's Encrypt certificates.
+
+All of this (every Linux VM to every OpenBSD box) will be connected via WireGuard tunnels, keeping everything private and secure. There will be 6 WireGuard tunnels (3 k3s nodes times two OpenBSD VMs).
+
+So, when I want to access a service running in k3s, I will hit an external DNS endpoint (with the authoritative DNS servers being the OpenBSD boxes). The DNS will resolve to the master OpenBSD VM (see my KISS highly-available with OpenBSD blog post), and from there, the relayd process (with a Let's Encrypt certificate—see my Let's Encrypt with OpenBSD and Rex blog post) will accept the TCP connection and forward it through the WireGuard tunnel to a reachable node port of one of the k3s nodes, thus serving the traffic.
+
+ +
> ~ curl https://f3s.foo.zone
+<html><body><h1>It works!</h1></body></html>
+
+> ~ curl https://www.f3s.foo.zone
+<html><body><h1>It works!</h1></body></html>
+
+> ~ curl https://standby.f3s.foo.zone
+<html><body><h1>It works!</h1></body></html>
+
+
+This is how it works in relayd.conf on OpenBSD:
+
+

OpenBSD relayd configuration


+
+The OpenBSD edge relays keep the Kubernetes-facing addresses for the f3s ingress endpoints in a shared backend table so TLS traffic for every f3s hostname lands on the same pool of k3s nodes (pointing to the WireGuard IP addresses of those nodes - remember, they are running locally in my LAN, wheras the OpenBSD edge relays operate in the public internet):
+
+
+table <f3s> {
+  192.168.2.120
+  192.168.2.121
+  192.168.2.122
+}
+
+
+Inside the http protocol "https" block each public hostname gets its Let's Encrypt certificate and is matched to that backend table. Besides the primary trio, every service-specific hostname (anki, bag, flux, audiobookshelf, gpodder, radicale, vault, syncthing, uprecords) and their www / standby aliases reuse the same pool so new apps can go live just by publishing an ingress rule, whereas they will all map to a service running in k3s:
+
+
+http protocol "https" {
+    tls keypair f3s.foo.zone
+    tls keypair www.f3s.foo.zone
+    tls keypair standby.f3s.foo.zone
+    tls keypair anki.f3s.foo.zone
+    tls keypair www.anki.f3s.foo.zone
+    tls keypair standby.anki.f3s.foo.zone
+    tls keypair bag.f3s.foo.zone
+    tls keypair www.bag.f3s.foo.zone
+    tls keypair standby.bag.f3s.foo.zone
+    tls keypair flux.f3s.foo.zone
+    tls keypair www.flux.f3s.foo.zone
+    tls keypair standby.flux.f3s.foo.zone
+    tls keypair audiobookshelf.f3s.foo.zone
+    tls keypair www.audiobookshelf.f3s.foo.zone
+    tls keypair standby.audiobookshelf.f3s.foo.zone
+    tls keypair gpodder.f3s.foo.zone
+    tls keypair www.gpodder.f3s.foo.zone
+    tls keypair standby.gpodder.f3s.foo.zone
+    tls keypair radicale.f3s.foo.zone
+    tls keypair www.radicale.f3s.foo.zone
+    tls keypair standby.radicale.f3s.foo.zone
+    tls keypair vault.f3s.foo.zone
+    tls keypair www.vault.f3s.foo.zone
+    tls keypair standby.vault.f3s.foo.zone
+    tls keypair syncthing.f3s.foo.zone
+    tls keypair www.syncthing.f3s.foo.zone
+    tls keypair standby.syncthing.f3s.foo.zone
+    tls keypair uprecords.f3s.foo.zone
+    tls keypair www.uprecords.f3s.foo.zone
+    tls keypair standby.uprecords.f3s.foo.zone
+
+    match request quick header "Host" value "f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "anki.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.anki.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.anki.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "bag.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.bag.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.bag.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "flux.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.flux.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.flux.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "audiobookshelf.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.audiobookshelf.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.audiobookshelf.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "gpodder.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.gpodder.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.gpodder.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "radicale.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.radicale.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.radicale.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "vault.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.vault.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.vault.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "syncthing.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.syncthing.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.syncthing.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "uprecords.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "www.uprecords.f3s.foo.zone" forward to <f3s>
+    match request quick header "Host" value "standby.uprecords.f3s.foo.zone" forward to <f3s>
+}
+
+
+Both IPv4 and IPv6 listeners reuse the same protocol definition, making the relay transparent for dual-stack clients while still health checking every k3s backend before forwarding traffic over WireGuard:
+
+
+relay "https4" {
+    listen on 46.23.94.99 port 443 tls
+    protocol "https"
+    forward to <f3s> port 80 check tcp
+}
+
+relay "https6" {
+    listen on 2a03:6000:6f67:624::99 port 443 tls
+    protocol "https"
+    forward to <f3s> port 80 check tcp
+}
+
+
+In practice, that means relayd terminates TLS with the correct certificate, keeps the three WireGuard-connected backends in rotation, and ships each request to whichever bhyve VM answers first.
+
+

Deploying the private Docker image registry


+
+As not all Docker images I want to deploy are available on public Docker registries and as I also build some of them by myself, there is the need of a private registry.
+
+All manifests for the f3s stack live in my configuration repository:
+
+codeberg.org/snonux/conf/f3s
+
+Within that repo, the examples/conf/f3s/registry/ directory contains the Helm chart, a Justfile, and a detailed README. Here's the condensed walkthrough I used to roll out the registry with Helm.
+
+

Prepare the NFS-backed storage


+
+Create the directory that will hold the registry blobs on the NFS share (I ran this on r0, but any node that exports /data/nfs/k3svolumes works):
+
+ +
[root@r0 ~]# mkdir -p /data/nfs/k3svolumes/registry
+
+
+

Install (or upgrade) the chart


+
+Clone the repo (or pull the latest changes) on a workstation that has helm configured for the cluster, then deploy the chart. The Justfile wraps the commands, but the raw Helm invocation looks like this:
+
+ +
$ git clone https://codeberg.org/snonux/conf/f3s.git
+$ cd conf/f3s/examples/conf/f3s/registry
+$ helm upgrade --install registry ./helm-chart --namespace infra --create-namespace
+
+
+Helm creates the infra namespace if it does not exist, provisions a PersistentVolume/PersistentVolumeClaim pair that points at /data/nfs/k3svolumes/registry, and spins up a single registry pod exposed via the docker-registry-service NodePort (30001). Verify everything is up before continuing:
+
+ +
$ kubectl get pods --namespace infra
+NAME                               READY   STATUS    RESTARTS      AGE
+docker-registry-6bc9bb46bb-6grkr   1/1     Running   6 (53d ago)   54d
+
+$ kubectl get svc docker-registry-service -n infra
+NAME                      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
+docker-registry-service   NodePort   10.43.141.56   <none>        5000:30001/TCP   54d
+
+
+

Allow nodes and workstations to trust the registry


+
+The registry listens on plain HTTP, so both Docker daemons on workstations and the k3s nodes need to treat it as an insecure registry. That's fine for my personal needs, as:
+
+
    +
  • I don't store any secrets in the images
    • +
  • I access the registry this way only via my LAN
    • +
  • I may will change it later on...
    • +

+On my Fedora workstation where I build images:
+
+ +
$ cat <<"EOF" | sudo tee /etc/docker/daemon.json >/dev/null
+{
+  "insecure-registries": [
+    "r0.lan.buetow.org:30001",
+    "r1.lan.buetow.org:30001",
+    "r2.lan.buetow.org:30001"
+  ]
+}
+EOF
+$ sudo systemctl restart docker
+
+
+On each k3s node, make registry.lan.buetow.org resolve locally and point k3s at the NodePort:
+
+ +
$ for node in r0 r1 r2; do
+>   ssh root@$node "echo '127.0.0.1 registry.lan.buetow.org' >> /etc/hosts"
+> done
+
+$ for node in r0 r1 r2; do
+> ssh root@$node "cat <<'EOF' > /etc/rancher/k3s/registries.yaml
+mirrors:
+  "registry.lan.buetow.org:30001":
+    endpoint:
+      - "http://localhost:30001"
+EOF
+systemctl restart k3s"
+> done
+
+
+Thanks to the relayd configuration earlier in the post, the external hostnames (f3s.foo.zone, etc.) can already reach NodePort 30001, so publishing the registry later to the outside world is just a matter of wiring the DNS the same way as the ingress hosts. But by default, that's not enabled for now due to security reasons.
+
+

Pushing and pulling images


+
+Tag any locally built image with one of the node IPs on port 30001, then push it. I usually target whichever node is closest to me, but any of the three will do:
+
+ +
$ docker tag my-app:latest r0.lan.buetow.org:30001/my-app:latest
+$ docker push r0.lan.buetow.org:30001/my-app:latest
+
+
+Inside the cluster (or from other nodes), reference the image via the service name that Helm created:
+
+
+image: docker-registry-service:5000/my-app:latest
+
+
+You can test the pull path straight away:
+
+ +
$ kubectl run registry-test \
+>   --image=docker-registry-service:5000/my-app:latest \
+>   --restart=Never -n test --command -- sleep 300
+
+
+If the pod pulls successfully, the private registry is ready for use by the rest of the workloads. Note, that the commands above actually don't work, they are only for illustration purpose mentioned here.
+
+

Example: Anki Sync Server from the private registry


+
+One of the first workloads I migrated onto the k3s cluster after standing up the registry was my Anki sync server. The configuration repo ships everything in examples/conf/f3s/anki-sync-server/: a Docker build context plus a Helm chart that references the freshly built image.
+
+

Build and push the image


+
+The Dockerfile lives under docker-image/ and takes the Anki release to compile as an ANKI_VERSION build argument. The accompanying Justfile wraps the steps, but the raw commands look like this:
+
+ +
$ cd conf/f3s/examples/conf/f3s/anki-sync-server/docker-image
+$ docker build -t anki-sync-server:25.07.5b --build-arg ANKI_VERSION=25.07.5 .
+$ docker tag anki-sync-server:25.07.5b \
+    r0.lan.buetow.org:30001/anki-sync-server:25.07.5b
+$ docker push r0.lan.buetow.org:30001/anki-sync-server:25.07.5b
+
+
+Because every k3s node treats registry.lan.buetow.org:30001 as an insecure mirror (see above), the push succeeds regardless of which node answers. If you prefer the shortcut, just f3s in that directory performs the same build/tag/push sequence.
+
+

Create the Anki secret and storage on the cluster


+
+The Helm chart expects the services namespace, a pre-created NFS directory, and a Kubernetes secret that holds the credentials the upstream container understands:
+
+ +
$ ssh root@r0 "mkdir -p /data/nfs/k3svolumes/anki-sync-server/anki_data"
+$ kubectl create namespace services
+$ kubectl create secret generic anki-sync-server-secret \
+    --from-literal=SYNC_USER1='paul:SECRETPASSWORD' \
+    -n services
+
+
+If the services namespace already exists, you can skip that line or let Kubernetes tell you the namespace is unchanged.
+
+

Deploy the chart


+
+With the prerequisites in place, install (or upgrade) the chart. It pins the container image to the tag we just pushed and mounts the NFS export via a PersistentVolume/PersistentVolumeClaim pair:
+
+ +
$ cd ../helm-chart
+$ helm upgrade --install anki-sync-server . -n services
+
+
+Helm provisions everything referenced in the templates:
+
+
+containers:
+- name: anki-sync-server  image: registry.lan.buetow.org:30001/anki-sync-server:25.07.5b
+  volumeMounts:
+  - name: anki-data
+    mountPath: /anki_data
+
+
+Once the release comes up, verify that the pod pulled the freshly pushed image and that the ingress we configured earlier resolves through relayd just like the Apache example.
+
+ +
$ kubectl get pods -n services
+$ kubectl get ingress anki-sync-server-ingress -n services
+$ curl https://anki.f3s.foo.zone/health
+
+
+All of this runs solely on first-party images that now live in the private registry, proving the full flow from local bild to WireGuard-exposed service.
+
+

NFSv4 UID mapping for Postgres-backed (and other) apps


+
+NFSv4 only sees numeric user and group IDs, so the postgres account created inside the container must exist with the same UID/GID on the Kubernetes worker and on the FreeBSD NFS servers. Otherwise the pod starts with UID 999, the export sees it as an unknown anonymous user, and Postgres fails to initialise its data directory.
+
+To verify things line up end-to-end I run id in the container and on the hosts:
+
+ +
> ~ kubectl exec -n services deploy/miniflux-postgres -- id postgres
+uid=999(postgres) gid=999(postgres) groups=999(postgres)
+
+[root@r0 ~]# id postgres
+uid=999(postgres) gid=999(postgres) groups=999(postgres)
+
+paul@f0:~ % doas id postgres
+uid=999(postgres) gid=99(postgres) groups=999(postgres)
+
+
+The Rocky Linux workers get their matching user with plain useradd/groupadd (repeat on r0, r1, and r2):
+
+ +
[root@r0 ~]# groupadd --gid 999 postgres
+[root@r0 ~]# useradd --uid 999 --gid 999 \
+                --home-dir /var/lib/pgsql \
+                --shell /sbin/nologin postgres
+
+
+FreeBSD uses pw, so on each NFS server (f0, f1, f2) I created the same account and disabled shell access:
+
+ +
paul@f0:~ % doas pw groupadd postgres -g 999
+paul@f0:~ % doas pw useradd postgres -u 999 -g postgres \
+                -d /var/db/postgres -s /usr/sbin/nologin
+
+
+Once the UID/GID exist everywhere, the Miniflux chart in examples/conf/f3s/miniflux deploys cleanly. The chart provisions both the application and its bundled Postgres database, mounts the exported directory, and builds the DSN at runtime. The important bits live in helm-chart/templates/persistent-volumes.yaml and deployment.yaml:
+
+
+# Persistent volume lives on the NFS export
+hostPath:
+  path: /data/nfs/k3svolumes/miniflux/data
+  type: Directory
+...
+containers:
+- name: miniflux-postgres
+  image: postgres:17
+  volumeMounts:
+  - name: miniflux-postgres-data
+    mountPath: /var/lib/postgresql/data
+
+
+Follow the README beside the chart to create the secrets and the target directory:
+
+ +
$ cd examples/conf/f3s/miniflux/helm-chart
+$ mkdir -p /data/nfs/k3svolumes/miniflux/data
+$ kubectl create secret generic miniflux-db-password \
+    --from-literal=fluxdb_password='YOUR_PASSWORD' -n services
+$ kubectl create secret generic miniflux-admin-password \
+    --from-literal=admin_password='YOUR_ADMIN_PASSWORD' -n services
+$ helm upgrade --install miniflux . -n services --create-namespace
+
+
+And to verify it's all up:
+
+
+$ kubectl get all --namespace=services | grep mini
+pod/miniflux-postgres-556444cb8d-xvv2p   1/1     Running   0             54d
+pod/miniflux-server-85d7c64664-stmt9     1/1     Running   0             54d
+service/miniflux                   ClusterIP   10.43.47.80     <none>        8080/TCP             54d
+service/miniflux-postgres          ClusterIP   10.43.139.50    <none>        5432/TCP             54d
+deployment.apps/miniflux-postgres   1/1     1            1           54d
+deployment.apps/miniflux-server     1/1     1            1           54d
+replicaset.apps/miniflux-postgres-556444cb8d   1         1         1       54d
+replicaset.apps/miniflux-server-85d7c64664     1         1         1       54d
+
+
+Or from the repository root I simply run:
+
+

Helm charts currently in service


+
+These are the charts that already live under examples/conf/f3s and run on the cluster today (and I'll keep adding more as new services graduate into production):
+
+
    +
  • anki-sync-server — custom-built image served from the private registry, stores decks on /data/nfs/k3svolumes/anki-sync-server/anki_data, and authenticates through the anki-sync-server-secret.
    • +
  • audiobookshelf — media streaming stack with three hostPath mounts (config, audiobooks, podcasts) so the library survives node rebuilds.
    • +
  • example-apache — minimal HTTP service I use for smoke-testing ingress and relayd rules.
    • +
  • example-apache-volume-claim — Apache plus PVC variant that exercises NFS-backed storage for walkthroughs like the one earlier in this post.
    • +
  • miniflux — the Postgres-backed feed reader described above, wired for NFSv4 UID mapping and per-release secrets.
    • +
  • opodsync — podsync deployment with its data directory under /data/nfs/k3svolumes/opodsync/data.
    • +
  • radicale — CalDAV/CardDAV (and gpodder) backend with separate collections and auth volumes.
    • +
  • registry — the plain-HTTP Docker registry exposed on NodePort 30001 and mirrored internally as registry.lan.buetow.org:30001.
    • +
  • syncthing — two-volume setup for config and shared data, fronted by the syncthing.f3s.foo.zone ingress.
    • +
  • wallabag — read-it-later service with persistent data and images directories on the NFS export.
    • +

+I hope you enjoyed this walkthrough. In the next part of this series, I will likely tackle monitoring, backup, or observability. I haven't fully decided yet which topic to cover next, so stay tuned!
+
+Other *BSD-related posts:
+
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)
+2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
+2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-04-01 KISS high-availability with OpenBSD
+2024-01-13 One reason why I love OpenBSD
+2022-10-30 Installing DTail on OpenBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+
+E-Mail your comments to paul@nospam.buetow.org
+
+Back to the main site
+
+ + Bash Golf Part 4 @@ -1291,6 +2372,7 @@ content = "{CODE}" 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments

f3s logo

@@ -2094,7 +3176,7 @@ ifconfig_re0_alias0="inet vhid 1 pass testpass alias 192.1 Next, update /etc/hosts on all nodes (f0, f1, f2, r0, r1, r2) to resolve the VIP hostname:

-192.168.1.138 f3s-storage-ha f3s-storage-ha.lan f3s-storage-ha.lan.buetow.org
+192.168.2.138 f3s-storage-ha f3s-storage-ha.wg0 f3s-storage-ha.wg0.wan.buetow.org

This allows clients to connect to f3s-storage-ha regardless of which physical server is currently the MASTER.
@@ -2850,7 +3932,7 @@ http://www.gnu.org/software/src-highlite --> clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1) # For persistent mount, add to /etc/fstab: -127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0 +127.0.0.1:/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev,soft,timeo=10,retrans=2,intr 0 0
Note: The mount uses localhost (127.0.0.1) because stunnel is listening locally and forwarding the encrypted traffic to the remote server.
@@ -3128,10 +4210,13 @@ Jul 06 10:Both technologies could run on top of our encrypted ZFS volumes, combining ZFS's data integrity and encryption features with distributed storage capabilities. This would be particularly interesting for workloads that need either S3-compatible APIs (MinIO) or transparent distributed POSIX storage (MooseFS). What about Ceph and GlusterFS? Unfortunately, there doesn't seem to be great native FreeBSD support for them. However, other alternatives also appear suitable for my use case.


-I'm looking forward to the next post in this series, where we will set up k3s (Kubernetes) on the Linux VMs.
+Read the next post of this series:
+
+f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments

Other *BSD-related posts:

+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)
2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
@@ -4194,6 +5279,7 @@ Jul 06 10:2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network (You are currently reading this)
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments

f3s logo

@@ -5174,6 +6260,7 @@ peer: 2htXdNcxzpI2FdPDJy4T4VGtm1wpMEQu1AkQHjNY6F8=
Other *BSD-related posts:

+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network (You are currently reading this)
2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
@@ -5755,6 +6842,7 @@ __ejm\___/________dwb`---`______________________ 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)
2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments

f3s logo

@@ -6331,6 +7419,7 @@ Apr 4 23: Other *BSD-related posts:

+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)
@@ -7056,6 +8145,7 @@ This is perl, v5.8.8 built gemini://foo.zone/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi 2024-12-02T23:48:21+02:00 @@ -8059,7 +9150,7 @@ Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded This is the second blog post about my f3s series for my self-hosting demands in my home lab. f3s? The 'f' stands for FreeBSD, and the '3s' stands for k3s, the Kubernetes distribution I will use on FreeBSD-based physical machines.
- f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+

f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation



Published at 2024-12-02T23:48:21+02:00

@@ -8075,6 +9166,7 @@ Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
+2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments

f3s logo

@@ -8085,6 +9177,7 @@ Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded

Table of Contents



- - - - 'Software Developmers Career Guide and Soft Skills' book notes - - gemini://foo.zone/gemfeed/2023-07-17-career-guide-and-soft-skills-book-notes.gmi - 2023-07-17T04:56:20+03:00 - - Paul Buetow aka snonux - paul@dev.buetow.org - - These notes are of two books by 'John Sommez' I found helpful. I also added some of my own keypoints to it. These notes are mainly for my own use, but you might find them helpful, too. - -
-

"Software Developmers Career Guide and Soft Skills" book notes


-
-Published at 2023-07-17T04:56:20+03:00
-
-These notes are of two books by "John Sommez" I found helpful. I also added some of my own keypoints to it. These notes are mainly for my own use, but you might find them helpful, too.
-
-
-         ,..........   ..........,
-     ,..,'          '.'          ',..,
-    ,' ,'            :            ', ',
-   ,' ,'             :             ', ',
-  ,' ,'              :              ', ',
- ,' ,'............., : ,.............', ',
-,'  '............   '.'   ............'  ',
- '''''''''''''''''';''';''''''''''''''''''
-                    '''
-
-
-

Table of Contents


-
-
-

Improve


-
-

Always learn new things


-
-When you learn something new, e.g. a programming language, first gather an overview, learn from multiple sources, play around and learn by doing and not consuming and form your own questions. Don't read too much upfront. A large amount of time is spent in learning technical skills which were never use. You want to have a practical set of skills you are actually using. You need to know 20 percent to get out 80 percent of the results.
-
-
    -
  • Learn a technology with a goal, e.g. implement a tool. Practice practise practice.
    • -
  • "I know X can do Y, I don't know exactly how, but I can look it up."
    • -
  • Read what experts are writing, for example follow blogs. Stay up to date and spent half an hour per day trading blogs and books.
    • -
  • Pick an open source application, read the code and try to understand it to get a feel of the syntax of the programming language.
    • -
  • Understand, that the standard library makes you a much better programmer.
    • -
  • Self learning is the top skill a programmer can have and is also useful in other aspects in your life.
    • -
  • Keep learning skills every day. Code every day. Don't be overconfident for job security. Read blogs, read books.
    • -
  • If you want to learn, then do it by exploring. Also teach what you learned (for example write a blog post or hold a presentation).
    • -

-Fake it until you make it. But be honest about your abilities or lack of. There is however only time between now and until you make it. Refer to your abilities to learn.
-
-Boot camps: The advantage of a boot camp is to pragmatically learn things fast. We almost always overestimate what we can do in a day. Especially during boot camps. Connect to others during the boot camps
-
-

Set goals


-
-Your own goals are important but the manager also looks at how the team performs and how someone can help the team perform better. Check whether you are on track with your goals every 2 weeks in order to avoid surprises for the annual review. Make concrete goals for next review. Track and document your progress. Invest in your education. Make your goals known. If you want something, then ask for it. Nobody but you knows what you want.
-
-

Ratings


-
-That's a trap: If you have to rate yourself, that's a trap. That never works in an unbiased way. Rate yourself always the best way but rate your weakest part as high as possible minus one point. Rate yourself as good as you can otherwise. Nobody is putting for fun a gun on his own head.
-
-
    -
  • Don't do peer rating, it can fire back on you. What if the colleague becomes your new boss?
    • -
  • Cooperate rankings are unfortunately HR guidelines and politics and only mirror a little your actual performance.
    • -

-

Promotions


-
-The most valuable employees are the ones who make themselves obsolete and automate all away. Keep a safety net of 3 to 6 months of finances. Safe at least 10 percent of your earnings. Also, if you make money it does not mean that you have to spent more money. Is a new car better than a used car which both can bring you from A to B? Liability vs assets.
-
-
    -
  • Raise or promotion, what's better? Promotion is better as money will follow anyway then.
    • -
  • Take projects no-one wants and make them shine. A promotion will follow.
    • -
  • A promotion is not going to come to you because you deserve it. You have to hunt and ask for it.
    • -
  • Track all kudos (e.g. ask for emails from your colleagues).
    • -
  • Big corporations HRs don't expect a figjit. That's why it's so important to keep track of your accomplishments and kudos'.
    • -
  • If you want a raise be specific how much and know to back your demands. Don't make a thread and no ultimatums.
    • -
  • Best way for a promotion is to switch jobs. You can even switch back with a better salary.
    • -

-

Finish things


-
-Hard work is necessary for accomplish results. However, work smarter not harder. Furthermore, working smart is not a substitute for working hard. Work both, hard and smart.
-
-
    -
  • Learn to finish things without motivation. Things will pay off when you stick to stuff and eventually motivation can also come back.
    • -
  • You will fail if you don't plan realistically. Set also a schedule and follow to it as of life depends on it.
    • -
  • Advances come only of you give more than asked. Consistency, commitment and knowing what you need to do is more key than hard work.
    • -
  • Any action is better than no action. If you get stuck you have gained nothing.
    • -
  • You need to know the unknowns. Identify as many unknown not known things as possible.
    • -

-Hard vs fun: Both engage the brain (video games vs work). Some work is hard and other is easy. Hard work is boring. The harsh truth is you have to put in hard and boring work in order to accomplish and be successful. Work won't be always boring though, as joy will follow with mastery.
-
-Defeat is finally give up. Failure is the road to success, embrace it. Failure does not define you but how you respond to it. Events don't make your unhappy, but how you react to events do.
-
-

Expand the empire


-
-The larger your empire is, the larger your circle of influence is. The larger the circle of influence is, the more opportunities you have.
-
-
    -
  • Do the dirty work if you want to expand the empire. That's there the opportunities are.
    • -
  • SCRUM often fails due to the lack to commitment. The backlog just becomes a wish to get completed.
    • -
  • Apply work on your quality standards. Don't cross the line of compromise. Always improve your skills. Never be happy being good enough.
    • -

-Become visible, keep track that you accomplishments. E.g. write a weekly summary. Do presentations, be seen. Learn new things and share your learnings. Be the problem solver and not the blamer.
-
-

Be pragmatic and also manage your time


-
-Make use of time boxing via the Pomodoro technique: Set a target of rounds and track the rounds. That give you exact focused work time. That's really the trick. For example set a goal of 6 daily pomodores.
-
-
    -
  • Every time you do something question why does it make sense be pragmatic and don't follow because it is best practice.
    • -
  • You can also apply the time boxing technique (Cal Newport) for focused deep work.
    • -

-You should feel good of the work done even if you don't finished the task. You will feel good about pomodoro wise even you don't finish the task on hand yet. Helps you to enjoy time off more. Working longer may not sell anything.
-
-

The quota system


-
-Defined quota of things done. E.g. N runs per week or M Blog posts per month or O pomodoros per week. This helps with consistency. Truly commit to these quotas. Failure is not an option. Start with small commitments. Don't commit to something you can't fulfill otherwise you set yourself up for failure.
-
-
    -
  • Why does the quota System work? Slow and consistent pace is the key. It also overcomes willpower weaknesses as goals are preset.
    • -
  • Internal motivation is more important over external motivation. Check out Daniels book drive.
    • -
  • Multitasking: Batching is effective. E.g. emails twice daily at pre-set times..
    • -

-

Don't waste time


-
-The biggest time waster is TV watching. The TV is programming you. It's insane that Americans watch so much TV as they work full time. Schedule one show at a time and watch it when you want to watch it. Most movies are crap anyways. The good movies will come to you as people will talk about them.
-
-
    -
  • Social media is time waster as well. Schedule your Social Media times. For example be on Facebook only for max one hour on Saturdays.
    • -
  • Meetings can waste time as well. Simply don't go to them. Try to cancel meeting if it can be dealt with via email.
    • -
  • Enjoying things is not a waste of time. E.g. you could still play a game once in a while. It is important not to cut away all you enjoy from your life.
    • -

-

Habits


-
-Try to have as many good habits as possible. Start with easy habits, and make them a little bit more challenging over time. Set ankers and rewards. Over time the routines will become habits naturally.
-
-Habit stacking is effective, which is combining multiple habits at the same time. For example you can workout on a circular trainer while while watching a learning video on O'Reilly Safari Online while getting closer to your weekly step goal.
-
-
    -
  • We don't have control over our habits but our own routines.
    • -
  • Routines help to form the habits, though.
    • -

-

Work-life balance


-
-Avoid overwork hours. That's not as beneficial as you might think and comes only with very small rewards. Invest rather in yourself and not in your employer.
-
-
    -
  • Work-life balance is a myth. Make it so that you enjoy work and your personal life and not just personal life.
    • -
  • Maintain fewer but good relationships. As a reward, better and integrated your life will be.
    • -
  • Life in the present Moment. Make the best of every moment of your life.
    • -
  • Enjoy every aspect of your life. If you want to take away one thing from this book that is it.
    • -

-Use your most productive hours to work on you. Make that your priority. Take care of yourself a priority (E.g. do workouts or learn a new language). You can always workout 2 or 1 hour per day, but will you pay the price?
-
-

Mental health


-
-
    -
  • Friendships and positive thinking help to have and maintain better health, longer Life, better productivity and increased happiness.
    • -
  • Positive thinking can be trained and be a habit. Read the book "The Power of Positive Thinking".
    • -
  • Stoicism helps. Meditation helps. Playing for fun helps too.
    • -

-Become the person you want to become (your self image). Program your brain unconsciously. Don't become the person other people want you to be. Embrace yourself, you are you.
-
-In most cases burnout is just an illusion. If you don't have motivation push through the wall. People usually don't pass the wall as they feel they are burned out. After pushing through the wall you will have the most fun, for example you will be able playing the guitar greatly.
-
-

Physical health


-
-Utilise a standing desk and treadmill (you could walk and type at the same time). Increase the incline in order to burn more calories. Even on the standing desk you burn more calories than sitting. When you use pomodoro then you can use the small breaks for push-ups (maybe won't do as good when you are in a fasted state).
-
-
    -
  • You can only do one thing, lose fat or gain muscles. Not both at the same time.
    • -
  • Train your strength by heavy lifting, but only with a very few repetitions (e.g. 5 max for each exercise, everything over this is body building).
    • -
  • If you want to increase the muscle mass use medium weights but lift them more often. If you want to increase your endurance lift light weights but with even more reps.
    • -
  • Avoid highly processed foods
    • -

-Intermittent fasting is an effective method to maintain weight and health. But it does not mean that you can only eat junk food in the feeding windows. Also, diet and nutrition is the most important for health and fitness. They make it also easier to stay focused and positive.
-
-

No drama


-
-Avoid drama at work. Where are humans there is drama. You can decide where to spent your energy in. But don't avoid conflict. Conflict is healthy in any kind of relationship. Be tactful and state your opinion. The goal is to find the best solution to the problem.
-
-Don't worry about other people what they do and don't do. You only worry about you. Shut up and get your own things done. But you could help to inspire a not working colleague.
-
-
    -
  • During an argument, take the opponent's position and see how your opinion changes.
    • -
  • If you they to convince someone else it's an argument. Of you try to find the best solution it is a good resolution.
    • -
  • If someone is hurting the team let the manager know but phrase it nicely.
    • -
  • How to get rid of a never ending talking person? Set up focus hours officially where you don't want to be interrupted. Present as if it is your defect that you get interrupted easily.
    • -
  • TOXIC PEOPLE: AVOID THEM. RUN.
    • -
  • Boss likes if you get shit done without getting asked all the time about things and also without drama.
    • -

-You have to learn how to work in a team. Be honest but tactful. It's not too be the loudest but about selling your ideas. Don't argue otherwise you won't sell anything. Be persuasive by finding the common ground. Or lead the colleagues to your idea and don't sell it upfront. Communicate clearly.
-
-

Personal brand


-
-
    -
  • Invest your value outside the company. Build your personal brand. Show how valuable you are, also to other companies. Become an asset.
    • -
  • Invest in your education. Make your goals known. If you want something ask for it (see also the sections about goals in this document).
    • -

-

Market yourself


-
-
    -
  • The best way to market yourself is to make you usable.
    • -
  • Create a brand. Decide your focus. Throw your name out as often as possible.
    • -

-Have a blog. Schedule your posts. Consistency beats every other factor. E.g. post once a month a new post. Find your voice, you don't have to sound academic. Keep writing, if you keep it long enough the rewards will be coming. Your own blog can take 5 years to take off. Most people give up too soon.
-
-
    -
  • Consistency of your blog is key. Also write quality content. Don't try to be a man of success but try to be a man of value.
    • -
  • Have an elevator pitch: "buetow.org - Having fun with computers!"
    • -
  • Have social media accounts, especially the ones which are more tech related.
    • -

-

Networking


-
-Ask people so they talk about themselves. They are not really interested in you. Use meetup.com to find groups you are interested and build up the network over time. Don't drink on social networking events even when others do. Talking to other people at events only has upsides. Just saying "hi" and introducing yourself is enough. What worse can happen? If the person rejects you so what, life goes on. Ask open questions and no "yes" and "no" questions. E.g.: "What is your story, why are you here?".
-
-

Public speaking


-
-Before your talk go on stage 10 minutes in advance. Introduce yourself to the front row people. During the talk they will smile at you and encourage you during your talk.
-
-
    -
  • Try at least 5 times before giving up public speaking. You can also start small, e.g. present a topic at work you are learning.
    • -
  • Practise your talk and timing. You can also record your practicing.
    • -

-Just do it. Just go to conferences. Even if you are not speaking. Sell your boss what you would learn and "this and that" and you would present the learnings to the team afterwards.
-
-

New job


-
-

For the interview


-
-
    -
  • Build up a network before the interview. E.g., follow and comment blogs. Or go to meet-ups and conferences. Join user groups.
    • -
  • Ask to touch base before the real interview and ask questions about the company. Do "pre-interviews".
    • -
  • Have a blog, a CV can only be 2 pages and an interview only can last only 2 hours. A blog helps you also to be a better communicator.
    • -

-If you are specialized then there is a better chance to get a fitting job. No one will hire a general lawyer if there are specialized lawyers available. Even if you are specialized, you will have a wide range of skills (T-shape knowledge).
-
-

Find the right type of company


-
-Not all companies are equal. They have individual cultures and guidelines.
-
-
    -
  • Startup: dynamic and larger impact. Many hats on.
    • -
  • Medium size companies: most stable ones. Not cutting edge technologies. No crazy working hours.
    • -
  • Large company: very established with a lot of structure however constant layoffs and restructurings. Less impact you can have. Complex politics.
    • -
  • Working for yourself: This is harder than you think, probably much harder.
    • -

-Work in a tech. company if you want to work on/with cutting edge technologies.
-
-

Apply for the new job


-
-Get a professional resume writer. Get referrals of writers and get samples from there. Get sufficient with algorithm and data structures interview questions. Cracking the coding interview book and blog
-
-
    -
  • Apply for each job with a specialised CV each. Each CV fits the job better.
    • -
  • Best get a job via a personal referral or inbound marketing. The latter is somehow rare.
    • -
  • Inbound marketing is for example someone responds to your blog and offers you a job.
    • -
  • Interview the interviewer. Be persistent.
    • -
  • Create creative looking resumes, see simple programmer website. Action-result style for a resume.
    • -

-Invest in your dress code as appearance masters. It does make sense to invest in your style. You could even hire a professional stylist (not my personal way though).
-
-

Negotiation


-
-
    -
  • Whoever names the number first loses. You don't know what someone else is expecting unless told. Low ball number may be an issue but you have to know the market.
    • -
  • Salary is not about what you need but what you are worth. Try to find out what you are worth.
    • -
  • Big tech companies have a pay scale. You can ask for this.
    • -
  • Don't tell your current salary. Only do one counter offer and say "If you do X then I commit today". Be tactful and not rude. Nobody wants to be taken advantage of. Also, don't be arrogant.
    • -
  • If the company wants to know your range, respond: "I would rather learn more about the job and compensation. You have a range in mind, correct?" Be brave and just pause here.
    • -
  • Otherwise, if the company refuses then say "if you tell me what the range is and although I am not yet sure yet what are my exact salary requirements are I can see if the range is of what I am looking for. If they absolute refuse give a high ball range you would expect and make it conditional to the overall compensation package. E.g. 70k to 100k depending on the compensation package. THE LOW END SHOULD BE YOUR REAL LOW END. Play a little bit of hardball here and be brave. Practise it.
    • -
  • Put 10 percent on top of the salary range into a counter offer.
    • -
  • Everything is negotiable, not only the salary.
    • -
  • Job markup rate: Check it regarding the recruitment rate negotiation.
    • -
  • Don't make a rushed decision based on deadlines. Make a fairly high counter offer shortly before deadline.
    • -
  • You should also cope with rejections while selling yourself. There is no such thing as job security.
    • -

-
    -
  • Never spilt the difference is the best book for learning negotiation techniques..
    • -

-

Leaving the old job


-
-When leaving a job make a clean and non personal as possible. Never complain and never explain. Don't worry about abandonment of the team. Everybody is replacement and you make a business decision. Don't threaten to quit as you are replaceable.
-
-

Other things


-
-
    -
  • As a leader lead by example and don't lead from the Eiffel tower.
    • -
  • As a leader you are responsible for the team. If the team fails then it's your fault only.
    • -

-

Testing


-
-Unit testing Vs regression testing: Unit tests test the smallest possible unit and get rewritten if the unit gets changed. It's like programming against a specification n. Regression tests test whether the software still works after the change. Now you know more than most software engineers.
-
-

Books to read


-
-
    -
  • Clean Code
    • -
  • Code Complete
    • -
  • Cracking the Interview - Lessons and Solutions.
    • -
  • Daniels Book "Drive" (about internal and external motivation)
    • -
  • God's degree (inventor of Dilbert)
    • -
  • Head first Design Patterns
    • -
  • How to win Friends and influence People
    • -
  • Never Split the Difference [X]
    • -
  • Structure and programming functional programs
    • -
  • The obstacle is the way [X]
    • -
  • The passionate programmer
    • -
  • The Power of Positive Thinking (Highly religious - I personally don't like it)
    • -
  • The Pragmatic Programmer [X]
    • -
  • The war of Art (to combat procrastination)
    • -
  • Willpower Instinct
    • -

-E-Mail your comments to paul@nospam.buetow.org :-)
-
-Other book notes of mine are:
-
-2025-06-07 "A Monk's Guide to Happiness" book notes
-2025-04-19 "When: The Scientific Secrets of Perfect Timing" book notes
-2024-10-24 "Staff Engineer" book notes
-2024-07-07 "The Stoic Challenge" book notes
-2024-05-01 "Slow Productivity" book notes
-2023-11-11 "Mind Management" book notes
-2023-07-17 "Software Developmers Career Guide and Soft Skills" book notes (You are currently reading this)
-2023-05-06 "The Obstacle is the Way" book notes
-2023-04-01 "Never split the difference" book notes
-2023-03-16 "The Pragmatic Programmer" book notes
-
Back to the main site
 -- cgit v1.2.3