update prompts

2 files changed, 343 insertions, 1 deletions

diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md
index 1e0601a..c067010 100644
--- a/prompts/skills/f3s/SKILL.md
+++ b/prompts/skills/f3s/SKILL.md
@@ -1,6 +1,6 @@
 ---
 name: f3s
-description: Reference skill for the f3s homelab—four Beelink S12 Pro hosts (f0/f1/f2/f3) running FreeBSD with Rocky Linux Bhyve VMs and a k3s Kubernetes cluster. f0/f1/f2 run r0/r1/r2 k3s nodes; f3 is standalone bhyve only (not part of k3s). Use when troubleshooting or making configuration decisions for the f3s setup.
+description: Reference skill for the f3s homelab—four Beelink S12 Pro hosts (f0/f1/f2/f3) running FreeBSD with Rocky Linux Bhyve VMs and a k3s Kubernetes cluster. f0/f1/f2 run r0/r1/r2 k3s nodes; f3 is standalone bhyve only (not part of k3s). Also includes four Raspberry Pi 3 nodes (pi0–pi3) running Rocky Linux 9. Use when troubleshooting or making configuration decisions for the f3s setup.
 ---
 
 # f3s Homelab Reference
@@ -43,6 +43,22 @@ Detailed reference documentation is in the `references/` subfolder:
 | earth | Fedora laptop (roaming) | — | 192.168.2.200 |
 | pixel7pro | Android (roaming) | — | 192.168.2.201 |
 | f3s-storage-ha | CARP VIP (f0/f1) | 192.168.1.138 | — |
+| pi0 | Raspberry Pi 3, Rocky Linux 9 | 192.168.1.125 | — |
+| pi1 | Raspberry Pi 3, Rocky Linux 9 | 192.168.1.126 | — |
+| pi2 | Raspberry Pi 3, Rocky Linux 9 | 192.168.1.127 | — |
+| pi3 | Raspberry Pi 3, Rocky Linux 9 | 192.168.1.128 | — |
+
+## Raspberry Pi Nodes
+
+Four Raspberry Pi 3 boards running Rocky Linux 9.2 (Blue Onyx) aarch64 from the SIG/AltArch image (`RockyLinuxRpi_9-latest.img.xz`). Each has:
+
+- User `paul` with passwordless sudo and SSH key auth
+- Static IP on eth0 via NetworkManager
+- Hostname `piN.lan.buetow.org`
+- Filesystem expanded with `rootfs-expand`
+- Default `rocky` user still present (password: `rockylinux`)
+- No GRUB — boots via Pi's native bootloader (`/boot/cmdline.txt`)
+- Custom RPi kernel from the `rockyrpi` repo
 
 ## Config Repository
 
diff --git a/prompts/skills/f3s/references/package-repos.md b/prompts/skills/f3s/references/package-repos.md
new file mode 100644
index 0000000..f8010be
--- /dev/null
+++ b/prompts/skills/f3s/references/package-repos.md
@@ -0,0 +1,326 @@
+# Package Repositories
+
+Custom FreeBSD and OpenBSD package repository served from k3s, allowing native package tools (`pkg` and `pkg_add`) to install custom-built packages.
+
+## Architecture
+
+- **nginx pod** in k3s `infra` namespace serves static files from a PV
+- Path prefixes: `/freebsd/` and `/openbsd/`
+- Accessible at `https://pkgrepo.f3s.buetow.org`
+- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod)
+- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile`
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) |
+| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest |
+| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD |
+| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` |
+| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends |
+| `frontends/Rexfile` (`gogios_install`) | Installs/updates gogios from the custom repo |
+| `packages/Makefile` | `make pkg` cross-compiles, packages, and uploads for both OSes |
+| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: pkg create + pkg repo + copy to PV |
+| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: pkg_create + signify signing |
+
+## PV Directory Structure
+
+```
+/data/nfs/k3svolumes/pkgrepo/
+  freebsd/
+    FreeBSD:15:amd64/
+      latest/
+        All/              # .pkg files
+        packagesite.pkg   # repo metadata (generated by `pkg repo`)
+        meta / meta.conf
+        data.pkg
+  openbsd/
+    7.8/
+      packages/
+        amd64/            # .tgz files
+```
+
+Hosted on NFS at f0 (`/data/nfs/k3svolumes/pkgrepo/`).
+
+## FreeBSD Client Setup (f0–f3)
+
+Custom repo is pre-configured on f0, f1, f2 (f3 offline at time of setup).
+Official FreeBSD repos remain active alongside the custom repo.
+
+File: `/usr/local/etc/pkg/repos/custom.conf`
+
+```
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+```
+
+### Using packages
+
+```sh
+doas pkg update
+doas pkg install <package-name>   # works for both official and custom packages
+```
+
+### Setting up a new FreeBSD host
+
+Pipe the config via stdin to avoid csh quoting issues (FreeBSD default shell is csh):
+
+```sh
+cat <<'REPO' | ssh -p 22 <host>.lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null'
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+REPO
+```
+
+## OpenBSD Client Setup (blowfish/fishfinger)
+
+Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`.
+Official OpenBSD packages still install normally via `/etc/installurl`.
+
+```sh
+export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/"
+```
+
+### Using packages
+
+```sh
+doas pkg_add <package>               # from custom repo (signed with signify)
+doas pkg_add <official-package>      # from official repo (still works)
+```
+
+### Setting up new OpenBSD hosts
+
+Two things needed:
+
+1. **Signify public key** — copy from an existing host to `/etc/signify/custom-pkg.pub`:
+   ```sh
+   scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/
+   scp /tmp/custom-pkg.pub rex@<newhost>:/tmp/
+   ssh rex@<newhost> "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub"
+   ```
+
+2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`:
+   ```sh
+   rex pkgrepo_setup   # adds PKG_PATH to /root/.profile on all frontends
+   ```
+
+The PKG_PATH must be updated when the OpenBSD version changes (currently 7.8).
+
+### Package signing
+
+OpenBSD packages are signed with `signify(1)` via `pkg_sign`:
+- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host)
+- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients
+- Signing happens automatically during `make pkg-openbsd` / `make pkg`
+- `pkg_add` verifies the signature against the public key — no `-D unsigned` needed
+
+## Packaging Workflow (Makefile)
+
+Build scripts live in `~/git/conf/packages/`, separate from individual project repos. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV.
+
+### Build and upload
+
+```sh
+cd ~/git/conf/packages
+
+# Both FreeBSD and OpenBSD
+make pkg NAME=gogios SRC=/home/paul/git/gogios \
+    COMMENT="Monitoring tool with email alerts and HTML status page" \
+    DESC="Gogios is a lightweight monitoring tool written in Go."
+
+# Single OS
+make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios
+make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios
+```
+
+### How it works
+
+**Single-binary packages (pure Go, cross-compilable):**
+1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`)
+2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD)
+3. Runs the script via SSH to package natively (`pkg create` / `pkg_create`)
+4. OpenBSD packages are signed with signify automatically
+5. FreeBSD repo metadata is regenerated with `pkg repo`
+6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/`
+
+**CGo packages (e.g. dtail with DataDog/zstd):**
+Cross-compilation from Linux to OpenBSD fails for CGo — needs a C cross-compiler.
+Solution: native build on the local OpenBSD VM. Source is sent via `git archive HEAD | ssh ... tar -x`
+(not `scp -r`) to avoid filling /tmp with build artifacts and test data.
+
+### Required variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `NAME` | Package name | `gogios` |
+| `SRC` | Go project root (must have `cmd/<NAME>/main.go` and `internal/version.go`) | `/home/paul/git/gogios` |
+
+### Optional variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `COMMENT` | `$(NAME)` | One-line description |
+| `DESC` | `$(NAME)` | Longer description |
+| `MAINTAINER` | `paul@buetow.org` | Maintainer email |
+| `WWW` | `https://buetow.org` | Project URL |
+| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC |
+
+### Version detection
+
+Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped.
+
+### Install/update on frontends via Rex
+
+```sh
+cd ~/git/conf/frontends
+rex gogios_install   # installs or updates gogios on blowfish + fishfinger
+rex gogios           # full setup (includes gogios_install + config + cron)
+```
+
+The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD).
+
+## DTail Package (multi-binary, OpenBSD only)
+
+DTail is a distributed log tailing tool with 6 binaries, config files, and an rc script.
+Unlike single-binary packages, it has a dedicated Makefile target and packaging script.
+
+### Build and upload
+
+```sh
+cd ~/git/conf/packages
+make dtail-openbsd
+```
+
+### What the package includes
+
+| File | Purpose |
+|------|---------|
+| `/usr/local/bin/dserver` | DTail server daemon |
+| `/usr/local/bin/dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | Client tools |
+| `/etc/dserver/dtail.json` | Server config (from `frontends/etc/dserver/dtail.json.tpl`) |
+| `/etc/rc.d/dserver` | RC script (from `frontends/etc/rc.d/dserver.tpl`) |
+| `/usr/local/bin/dserver-update-key-cache.sh` | SSH key cache helper for daily cron |
+
+### Rex integration
+
+```sh
+cd ~/git/conf/frontends
+rex dtail_install   # install/update dtail package from custom repo
+rex dtail           # full setup (install + service user + daily cron + start)
+```
+
+The `dtail` Rex task installs the package, creates the `_dserver` service user,
+adds the key cache script to daily cron, and ensures dserver is running.
+
+### Why a build VM?
+
+DTail depends on DataDog/zstd which requires CGo. Cross-compiling CGo from Linux
+to OpenBSD needs a C cross-compiler. Instead, a local OpenBSD QEMU/KVM VM builds
+natively — this also works for any future packages with C dependencies.
+
+## OpenBSD Build VM
+
+A minimal OpenBSD QEMU/KVM VM on the laptop (earth) for native compilation.
+Scripts in `packages/buildvm/`.
+
+### Initial setup (once)
+
+```sh
+cd ~/git/conf/packages/buildvm
+
+# Fetch signify keys from fishfinger
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec .
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub .
+
+./setup.sh       # downloads ISO, runs fully automated install (~5 min)
+./provision.sh   # installs Go, git, gmake, signify keys, SSH keys
+```
+
+`setup.sh` is fully automated — it calls `install-expect.exp` which drives the
+OpenBSD serial console installer via expect. No manual interaction needed.
+
+### Day-to-day use
+
+```sh
+make buildvm-start   # boot VM (~15s)
+make dtail-openbsd   # auto-starts VM if needed
+make buildvm-stop    # shut down when done
+```
+
+The VM is headless, SSH on `localhost:2222`, 1GB RAM, 2 CPUs, 4GB disk.
+Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`.
+
+### Automated installer notes (install-expect.exp)
+
+- The expect script is in a **separate file** (`install-expect.exp`) to avoid bash/expect quoting interactions — embedding expect in a bash heredoc causes password send failures
+- Serial console is activated via `set tty com0` at the OpenBSD boot prompt
+- Password prompts require `sleep 2` before `send` — `sleep 1` is not enough for the serial console to be ready
+- OpenBSD 7.8 added a new "Encrypt the root disk?" prompt before the partition layout
+- After CONGRATULATIONS, choose shell (`s`) instead of reboot — the CD is still attached and reboot re-boots the installer; configure wheel group via `chroot /mnt`
+- Username `build` is rejected by the installer ("not a usable loginname") — use `pbuild` instead
+- Source code is synced to the VM via `git archive HEAD | ssh ... tar -x` to avoid filling /tmp (full repo with benchmarks exceeds available space on 4GB disk)
+
+## Building Custom Packages
+
+### FreeBSD
+
+On any FreeBSD host, or via the Mage SCP-and-remote-execute pattern:
+
+```sh
+pkg create -M +MANIFEST -p plist -r stagedir -o output/All
+pkg repo output/   # regenerates repo metadata (unsigned)
+```
+
+Copy output to PV:
+```sh
+doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/
+```
+
+### OpenBSD
+
+On an OpenBSD host:
+
+```sh
+pkg_create \
+    -D COMMENT="Package description" \
+    -d descfile \
+    -f packing-list \
+    -B stagedir \
+    -p / \
+    output/package-name-1.0.tgz
+```
+
+Copy to PV via f0:
+```sh
+scp package.tgz f0.lan.buetow.org:/tmp/
+ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/"
+```
+
+## SSH Access for Package Tasks
+
+| Host | Access | Root | Notes |
+|------|--------|------|-------|
+| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local, default shell is csh |
+| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | Packages built here, then copied to f0 PV |
+| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger |
+
+## Notes
+
+- HTTP is always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://`
+- OpenBSD version in the path must match the host OS version (currently 7.8)
+- FreeBSD version follows the ABI naming convention (`FreeBSD:15:amd64`)
+- OpenBSD packages are signed with signify (key pair at `/etc/signify/custom-pkg.*`)
+- FreeBSD packages are unsigned (`signature_type: "NONE"`) — signing with RSA is possible but not yet configured
+- After adding/removing FreeBSD packages, always regenerate repo metadata with `pkg repo`
+- OpenBSD does not need a repo index — `pkg_add` fetches packages directly by name
+- FreeBSD hosts use csh as default shell — remote scripts must use `/bin/sh` explicitly
+- f3 is currently offline; configure its custom repo when it comes back online