From 3d87f7f82e2b672d9d35898d2a3d8b62c6687918 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sat, 11 Apr 2026 15:22:04 +0300 Subject: skills: split package repo docs out of f3s --- prompts/skills/f3s/SKILL.md | 3 +- prompts/skills/f3s/references/dtail.md | 89 ++++- prompts/skills/f3s/references/package-repos.md | 326 --------------- prompts/skills/pkgrepo/SKILL.md | 31 ++ prompts/skills/pkgrepo/references/package-repos.md | 435 +++++++++++++++++++++ 5 files changed, 543 insertions(+), 341 deletions(-) delete mode 100644 prompts/skills/f3s/references/package-repos.md create mode 100644 prompts/skills/pkgrepo/SKILL.md create mode 100644 prompts/skills/pkgrepo/references/package-repos.md diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md index 5757862..9aabebc 100644 --- a/prompts/skills/f3s/SKILL.md +++ b/prompts/skills/f3s/SKILL.md @@ -25,12 +25,13 @@ Detailed reference documentation is in the `references/` subfolder: - [Storage](references/storage.md) — ZFS (zdata), CARP, NFS over stunnel, zrepl replication - [k3s Setup](references/k3s-setup.md) — HA k3s cluster, etcd, node IPs, kubeconfig, ArgoCD - [Observability](references/observability.md) — Prometheus, Grafana, Loki, Alloy, Tempo -- [Package Repos](references/package-repos.md) — Custom FreeBSD/OpenBSD pkg repo served from k3s nginx pod - [Immich](references/immich.md) — Photo server deployment, job queue stats, troubleshooting - [Garage](references/garage.md) — Garage cluster, edge domain routing, S3 bucket/key workflow, troubleshooting - [DTail / dserver](references/dtail.md) — dserver: Pis **arm64** vs r0–r2 **amd64**, r-VM **root** + `root.authorized_keys` cache, firewalld **2222**, systemd timers - [dserver.d](references/dserver.d) — index: links to **Rocky r-VM DTail** subsection and full **dtail.md** +Package repository details were split into the sibling `pkgrepo` skill. Use `pkgrepo` for `pkgrepo.f3s.buetow.org`, repo layout, package publication, and client repo configuration. + ## Quick Reference: Host IPs | Host | Role | LAN IP | WireGuard IP | diff --git a/prompts/skills/f3s/references/dtail.md b/prompts/skills/f3s/references/dtail.md index bfee00a..5d6a2a8 100644 --- a/prompts/skills/f3s/references/dtail.md +++ b/prompts/skills/f3s/references/dtail.md @@ -10,8 +10,15 @@ Upstream install and examples live in the repo: `doc/installation.md`, `examples |-------|-----------|----------------|------------------| | **pi0–pi3** | Rocky Linux 9 **aarch64** (Raspberry Pi 3) | Cross-build **linux/arm64**, `nozstd` | `paul@piN.lan.buetow.org` | | **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` | +| **blowfish, fishfinger** | OpenBSD 7.8 **amd64** | Native OpenBSD package build | `rex@blowfish.buetow.org`, `rex@fishfinger.buetow.org` | -**Root login and key cache:** `examples/update_key_cache.sh.example` only scans `/home/*`. If clients connect as **root**, copy keys once (e.g. after install) and on key changes: +**Key cache filenames matter:** `examples/update_key_cache.sh.example` only scans `/home/*` and writes `/var/run/dserver/cache/USER.authorized_keys`. In this lab, DTail auth worked only after writing the exact cache filename for the login user: + +- **r0–r2**: `root.authorized_keys` +- **pi0–pi3**: `paul.authorized_keys` +- **blowfish, fishfinger**: `rex.authorized_keys` + +If clients connect as **root**, copy keys once (e.g. after install) and on key changes: ```bash cp /root/.ssh/authorized_keys /var/run/dserver/cache/root.authorized_keys @@ -19,6 +26,14 @@ chown dserver:dserver /var/run/dserver/cache/root.authorized_keys chmod 600 /var/run/dserver/cache/root.authorized_keys ``` +For the Pi nodes: + +```bash +cp /home/paul/.ssh/authorized_keys /var/run/dserver/cache/paul.authorized_keys +chown dserver:dserver /var/run/dserver/cache/paul.authorized_keys +chmod 600 /var/run/dserver/cache/paul.authorized_keys +``` + ## dserver on r0, r1, r2 (k3s Rocky VMs, amd64) These hosts are the **x86_64** guests on f0/f1/f2. SSH and VM background: [Rocky Linux VMs](rocky-linux-vms.md), **DTail subsection** (same content in short form): [DTail (dserver) on r0–r2](rocky-linux-vms.md#dtail-dserver-on-r0r2). Shortcut index file: [dserver.d](dserver.d). **Do not** install the Pi **arm64** binary here. @@ -30,14 +45,19 @@ These hosts are the **x86_64** guests on f0/f1/f2. SSH and VM background: [Rocky | Admin SSH (normal) | `ssh -p 22 root@rN.lan.buetow.org` (key-based; see Rocky VM doc) | | dserver port | **2222/tcp** (DTail clients); **22** remains sshd | -### Build the binary (on earth) +### Build the binaries (on earth) ```bash cd ~/git/dtail # your checkout of https://codeberg.org/snonux/dtail -CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver -# artifact: ./dserver (static amd64; copy as e.g. dserver-linux-amd64) + +# exact cross-builds for the Rocky VMs +for bin in dserver dtail dcat dgrep dmap dtailhealth; do + CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o "$bin-linux-amd64" ./cmd/$bin/main.go +done ``` +Use direct `go build -tags nozstd` when you need a deterministic cross-build artifact. Reusing `make dserver` after a local native build can leave an old binary in place because the target name is just `dserver`. + ### Install on each rN (run as root over SSH) 1. Copy to the VM, e.g. `scp -P 22 dserver root@r0.lan.buetow.org:/tmp/` and the files from `examples/` (see checklist below), or unpack a small staging dir on `/tmp/dtail-install/`. @@ -65,6 +85,22 @@ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver systemctl start dserver-update-keycache.service # populate cache once ``` + **Important:** the stock unit uses `WorkingDirectory=/var/run/dserver`. On Rocky and Pi hosts, `dserver` failed with `status=200/CHDIR` until the unit also recreated that tmpfs path at service start. Add: + + ```ini + RuntimeDirectory=dserver + RuntimeDirectoryMode=0755 + ExecStartPre=/usr/bin/mkdir -p /var/run/dserver/cache /var/run/dserver/log + ``` + + Then run: + + ```bash + systemctl daemon-reload + systemctl reset-failed dserver + systemctl start dserver + ``` + 5. **Root SSH key cache (required on r VMs)** The stock `update_key_cache.sh` only copies `/home/*/.ssh/authorized_keys`. **Root’s keys live in `/root`**, so mirror them explicitly (after any change to root’s authorized_keys): @@ -95,9 +131,9 @@ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver DTail uses the **same username** as normal SSH unless overridden. If dserver is set up for **root** only on r VMs, run as root or pass the client user your config expects. Example: ```bash -dcat --servers r0.lan,r1.lan,r2.lan /etc/hostname -# if your client user must be root for key + dtail.json to match: -# sudo dcat ... # or configure SSH user flags if the client supports them +dcat --plain --noColor --trustAllHosts --user root \ + --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org \ + --files /etc/fstab ``` Add **2222** host keys to `~/.ssh/known_hosts` the first time (interactive trust, or `ssh-keyscan -p 2222`). @@ -110,12 +146,10 @@ From a clone of the repo: cd ~/git/dtail # or your checkout # Raspberry Pi 4× — linux/arm64, static, no CGO zstd -CGO_ENABLED=0 GOOS=linux GOARCH=arm64 DTAIL_NO_ZSTD=yes make dserver -# produces ./dserver — keep a copy e.g. ./dserver-linux-arm64 +CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -tags nozstd -o dserver-linux-arm64 ./cmd/dserver/main.go # k3s VMs r0–r2 — linux/amd64 -CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver -# produces ./dserver — keep a copy e.g. ./dserver-linux-amd64 +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o dserver-linux-amd64 ./cmd/dserver/main.go ``` `DTAIL_NO_ZSTD=yes` sets the `nozstd` build tag so the binary does not link DataDog’s CGO zstd (required for static cross-compiles). **`.zst` log files are not supported** in that binary; gzip still works. @@ -128,7 +162,7 @@ Do this on **each** target (pi0–pi3 and/or r0–r2). Adjust **user** if you ar 2. **OS user**: `dserver` system account (`useradd -r -d /var/lib/dserver -s /sbin/nologin -U dserver`). 3. **Directories**: `/etc/dserver`, `/var/run/dserver` (tmpfs — recreated at boot; use systemd `RuntimeDirectory` in the example unit if you adopt it). 4. **Config**: `/etc/dserver/dtail.json` from `examples/dtail.json.example` — ensure **Server.Permissions.Users** includes every login name that will connect (e.g. `paul`, `root`). -5. **systemd**: `examples/dserver.service.example` → `/etc/systemd/system/dserver.service` — unit stays **disabled** by default; start with `systemctl start dserver`. +5. **systemd**: `examples/dserver.service.example` → `/etc/systemd/system/dserver.service` — unit stays **disabled** by default; start with `systemctl start dserver`. Add `RuntimeDirectory=dserver`, `RuntimeDirectoryMode=0755`, and `ExecStartPre=/usr/bin/mkdir -p /var/run/dserver/cache /var/run/dserver/log` so the tmpfs working directory exists before `dserver` starts. 6. **SSH host keys for clients**: dserver cannot read users’ `~/.ssh/authorized_keys` as user `dserver`. Use `examples/update_key_cache.sh.example` + `dserver-update-keycache.service` / `.timer` to mirror keys into `/var/run/dserver/cache/USER.authorized_keys`. 7. **firewalld**: allow **2222/tcp** (ping may work while TCP is blocked): @@ -149,12 +183,39 @@ Do this on **each** target (pi0–pi3 and/or r0–r2). Adjust **user** if you ar ## Client examples ```bash -dcat --servers pi0.lan,pi1.lan,pi2.lan,pi3.lan /etc/os-release -dcat --servers r0.lan,r1.lan,r2.lan /etc/hostname +dcat --plain --noColor --trustAllHosts --user paul \ + --servers pi0.lan.buetow.org,pi1.lan.buetow.org,pi2.lan.buetow.org,pi3.lan.buetow.org \ + --files /etc/fstab + +dcat --plain --noColor --trustAllHosts --user root \ + --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org \ + --files /etc/fstab ``` Use hostnames that resolve from where you run the client (often `*.lan.buetow.org`). +## Package-managed DTail + +For package-repo-backed DTail and other custom package repo tasks, use the sibling `pkgrepo` skill and its reference: + +- [Package Repositories](../../pkgrepo/references/package-repos.md) + +That skill now owns: + +- FreeBSD, OpenBSD, and Rocky client repo configuration +- custom repo layout, publication, and verification notes +- Rocky repo client configuration and RPM install flow +- OpenBSD DTail package build, publish, replace, and cache-refresh steps +- DTail package publication details that depend on the repo + +## Verified lab state + +On 2026-04-11 this setup was verified end-to-end with: + +- the full Linux binary set (`dserver`, `dtail`, `dcat`, `dgrep`, `dmap`, `dtailhealth`) installed on `r0`-`r2` and `pi0`-`pi3` +- `dserver` active and listening on `*:2222` on all seven hosts +- successful `dcat /etc/fstab` reads from earth using `--user root` for `r0`-`r2` and `--user paul` for `pi0`-`pi3` + ## k3s / r0–r2 notes - **amd64** binaries only; do not deploy the Pi **arm64** build there. diff --git a/prompts/skills/f3s/references/package-repos.md b/prompts/skills/f3s/references/package-repos.md deleted file mode 100644 index f8010be..0000000 --- a/prompts/skills/f3s/references/package-repos.md +++ /dev/null @@ -1,326 +0,0 @@ -# Package Repositories - -Custom FreeBSD and OpenBSD package repository served from k3s, allowing native package tools (`pkg` and `pkg_add`) to install custom-built packages. - -## Architecture - -- **nginx pod** in k3s `infra` namespace serves static files from a PV -- Path prefixes: `/freebsd/` and `/openbsd/` -- Accessible at `https://pkgrepo.f3s.buetow.org` -- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod) -- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile` - -## Key Files - -| File | Purpose | -|------|---------| -| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) | -| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest | -| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD | -| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` | -| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends | -| `frontends/Rexfile` (`gogios_install`) | Installs/updates gogios from the custom repo | -| `packages/Makefile` | `make pkg` cross-compiles, packages, and uploads for both OSes | -| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: pkg create + pkg repo + copy to PV | -| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: pkg_create + signify signing | - -## PV Directory Structure - -``` -/data/nfs/k3svolumes/pkgrepo/ - freebsd/ - FreeBSD:15:amd64/ - latest/ - All/ # .pkg files - packagesite.pkg # repo metadata (generated by `pkg repo`) - meta / meta.conf - data.pkg - openbsd/ - 7.8/ - packages/ - amd64/ # .tgz files -``` - -Hosted on NFS at f0 (`/data/nfs/k3svolumes/pkgrepo/`). - -## FreeBSD Client Setup (f0–f3) - -Custom repo is pre-configured on f0, f1, f2 (f3 offline at time of setup). -Official FreeBSD repos remain active alongside the custom repo. - -File: `/usr/local/etc/pkg/repos/custom.conf` - -``` -custom: { - url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest", - mirror_type: "NONE", - signature_type: "NONE", - enabled: yes -} -``` - -### Using packages - -```sh -doas pkg update -doas pkg install # works for both official and custom packages -``` - -### Setting up a new FreeBSD host - -Pipe the config via stdin to avoid csh quoting issues (FreeBSD default shell is csh): - -```sh -cat <<'REPO' | ssh -p 22 .lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null' -custom: { - url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest", - mirror_type: "NONE", - signature_type: "NONE", - enabled: yes -} -REPO -``` - -## OpenBSD Client Setup (blowfish/fishfinger) - -Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`. -Official OpenBSD packages still install normally via `/etc/installurl`. - -```sh -export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/" -``` - -### Using packages - -```sh -doas pkg_add # from custom repo (signed with signify) -doas pkg_add # from official repo (still works) -``` - -### Setting up new OpenBSD hosts - -Two things needed: - -1. **Signify public key** — copy from an existing host to `/etc/signify/custom-pkg.pub`: - ```sh - scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/ - scp /tmp/custom-pkg.pub rex@:/tmp/ - ssh rex@ "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub" - ``` - -2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`: - ```sh - rex pkgrepo_setup # adds PKG_PATH to /root/.profile on all frontends - ``` - -The PKG_PATH must be updated when the OpenBSD version changes (currently 7.8). - -### Package signing - -OpenBSD packages are signed with `signify(1)` via `pkg_sign`: -- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host) -- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients -- Signing happens automatically during `make pkg-openbsd` / `make pkg` -- `pkg_add` verifies the signature against the public key — no `-D unsigned` needed - -## Packaging Workflow (Makefile) - -Build scripts live in `~/git/conf/packages/`, separate from individual project repos. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV. - -### Build and upload - -```sh -cd ~/git/conf/packages - -# Both FreeBSD and OpenBSD -make pkg NAME=gogios SRC=/home/paul/git/gogios \ - COMMENT="Monitoring tool with email alerts and HTML status page" \ - DESC="Gogios is a lightweight monitoring tool written in Go." - -# Single OS -make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios -make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios -``` - -### How it works - -**Single-binary packages (pure Go, cross-compilable):** -1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`) -2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD) -3. Runs the script via SSH to package natively (`pkg create` / `pkg_create`) -4. OpenBSD packages are signed with signify automatically -5. FreeBSD repo metadata is regenerated with `pkg repo` -6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/` - -**CGo packages (e.g. dtail with DataDog/zstd):** -Cross-compilation from Linux to OpenBSD fails for CGo — needs a C cross-compiler. -Solution: native build on the local OpenBSD VM. Source is sent via `git archive HEAD | ssh ... tar -x` -(not `scp -r`) to avoid filling /tmp with build artifacts and test data. - -### Required variables - -| Variable | Description | Example | -|----------|-------------|---------| -| `NAME` | Package name | `gogios` | -| `SRC` | Go project root (must have `cmd//main.go` and `internal/version.go`) | `/home/paul/git/gogios` | - -### Optional variables - -| Variable | Default | Description | -|----------|---------|-------------| -| `COMMENT` | `$(NAME)` | One-line description | -| `DESC` | `$(NAME)` | Longer description | -| `MAINTAINER` | `paul@buetow.org` | Maintainer email | -| `WWW` | `https://buetow.org` | Project URL | -| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC | - -### Version detection - -Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped. - -### Install/update on frontends via Rex - -```sh -cd ~/git/conf/frontends -rex gogios_install # installs or updates gogios on blowfish + fishfinger -rex gogios # full setup (includes gogios_install + config + cron) -``` - -The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD). - -## DTail Package (multi-binary, OpenBSD only) - -DTail is a distributed log tailing tool with 6 binaries, config files, and an rc script. -Unlike single-binary packages, it has a dedicated Makefile target and packaging script. - -### Build and upload - -```sh -cd ~/git/conf/packages -make dtail-openbsd -``` - -### What the package includes - -| File | Purpose | -|------|---------| -| `/usr/local/bin/dserver` | DTail server daemon | -| `/usr/local/bin/dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | Client tools | -| `/etc/dserver/dtail.json` | Server config (from `frontends/etc/dserver/dtail.json.tpl`) | -| `/etc/rc.d/dserver` | RC script (from `frontends/etc/rc.d/dserver.tpl`) | -| `/usr/local/bin/dserver-update-key-cache.sh` | SSH key cache helper for daily cron | - -### Rex integration - -```sh -cd ~/git/conf/frontends -rex dtail_install # install/update dtail package from custom repo -rex dtail # full setup (install + service user + daily cron + start) -``` - -The `dtail` Rex task installs the package, creates the `_dserver` service user, -adds the key cache script to daily cron, and ensures dserver is running. - -### Why a build VM? - -DTail depends on DataDog/zstd which requires CGo. Cross-compiling CGo from Linux -to OpenBSD needs a C cross-compiler. Instead, a local OpenBSD QEMU/KVM VM builds -natively — this also works for any future packages with C dependencies. - -## OpenBSD Build VM - -A minimal OpenBSD QEMU/KVM VM on the laptop (earth) for native compilation. -Scripts in `packages/buildvm/`. - -### Initial setup (once) - -```sh -cd ~/git/conf/packages/buildvm - -# Fetch signify keys from fishfinger -scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec . -scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub . - -./setup.sh # downloads ISO, runs fully automated install (~5 min) -./provision.sh # installs Go, git, gmake, signify keys, SSH keys -``` - -`setup.sh` is fully automated — it calls `install-expect.exp` which drives the -OpenBSD serial console installer via expect. No manual interaction needed. - -### Day-to-day use - -```sh -make buildvm-start # boot VM (~15s) -make dtail-openbsd # auto-starts VM if needed -make buildvm-stop # shut down when done -``` - -The VM is headless, SSH on `localhost:2222`, 1GB RAM, 2 CPUs, 4GB disk. -Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`. - -### Automated installer notes (install-expect.exp) - -- The expect script is in a **separate file** (`install-expect.exp`) to avoid bash/expect quoting interactions — embedding expect in a bash heredoc causes password send failures -- Serial console is activated via `set tty com0` at the OpenBSD boot prompt -- Password prompts require `sleep 2` before `send` — `sleep 1` is not enough for the serial console to be ready -- OpenBSD 7.8 added a new "Encrypt the root disk?" prompt before the partition layout -- After CONGRATULATIONS, choose shell (`s`) instead of reboot — the CD is still attached and reboot re-boots the installer; configure wheel group via `chroot /mnt` -- Username `build` is rejected by the installer ("not a usable loginname") — use `pbuild` instead -- Source code is synced to the VM via `git archive HEAD | ssh ... tar -x` to avoid filling /tmp (full repo with benchmarks exceeds available space on 4GB disk) - -## Building Custom Packages - -### FreeBSD - -On any FreeBSD host, or via the Mage SCP-and-remote-execute pattern: - -```sh -pkg create -M +MANIFEST -p plist -r stagedir -o output/All -pkg repo output/ # regenerates repo metadata (unsigned) -``` - -Copy output to PV: -```sh -doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/ -``` - -### OpenBSD - -On an OpenBSD host: - -```sh -pkg_create \ - -D COMMENT="Package description" \ - -d descfile \ - -f packing-list \ - -B stagedir \ - -p / \ - output/package-name-1.0.tgz -``` - -Copy to PV via f0: -```sh -scp package.tgz f0.lan.buetow.org:/tmp/ -ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/" -``` - -## SSH Access for Package Tasks - -| Host | Access | Root | Notes | -|------|--------|------|-------| -| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local, default shell is csh | -| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | Packages built here, then copied to f0 PV | -| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger | - -## Notes - -- HTTP is always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://` -- OpenBSD version in the path must match the host OS version (currently 7.8) -- FreeBSD version follows the ABI naming convention (`FreeBSD:15:amd64`) -- OpenBSD packages are signed with signify (key pair at `/etc/signify/custom-pkg.*`) -- FreeBSD packages are unsigned (`signature_type: "NONE"`) — signing with RSA is possible but not yet configured -- After adding/removing FreeBSD packages, always regenerate repo metadata with `pkg repo` -- OpenBSD does not need a repo index — `pkg_add` fetches packages directly by name -- FreeBSD hosts use csh as default shell — remote scripts must use `/bin/sh` explicitly -- f3 is currently offline; configure its custom repo when it comes back online diff --git a/prompts/skills/pkgrepo/SKILL.md b/prompts/skills/pkgrepo/SKILL.md new file mode 100644 index 0000000..77de23e --- /dev/null +++ b/prompts/skills/pkgrepo/SKILL.md @@ -0,0 +1,31 @@ +--- +name: pkgrepo +description: Reference skill for the homelab package repositories behind pkgrepo.f3s.buetow.org. Covers FreeBSD pkg, OpenBSD pkg_add, Rocky Linux dnf repos, PV layout, nginx/ArgoCD wiring, DTail package publishing, and client repo configuration. Use when publishing packages, troubleshooting repository access or metadata, or deciding how packages should be installed from the custom repo. +--- + +# Package Repo Reference + +Use this skill when the task is specifically about the custom package repositories rather than the broader f3s homelab. + +## When to Use + +- Publishing or updating packages in `pkgrepo.f3s.buetow.org` +- Troubleshooting repo layout, metadata, or HTTP exposure +- Configuring FreeBSD, OpenBSD, or Rocky Linux clients to install from the custom repo +- Working on DTail package publishing and installation through the repo + +## Reference Files + +- [Package Repositories](references/package-repos.md) — repo layout, PV paths, nginx exposure, client config, and DTail package workflows + +## Scope + +This skill owns package repository details that used to live under `f3s`. + +Use `f3s` alongside this skill when the task depends on broader host-role or cluster context, especially: + +- `f0` as the FreeBSD NFS/PV host for `/data/nfs/k3svolumes/pkgrepo/` +- `fishfinger` and `blowfish` as the OpenBSD frontend hosts +- `r0-r2` as Rocky Linux x86_64 bhyve VMs +- `pi0-pi3` as Rocky Linux aarch64 Raspberry Pi nodes +- `earth` as the Fedora laptop used for package publication and verification diff --git a/prompts/skills/pkgrepo/references/package-repos.md b/prompts/skills/pkgrepo/references/package-repos.md new file mode 100644 index 0000000..d6c45c4 --- /dev/null +++ b/prompts/skills/pkgrepo/references/package-repos.md @@ -0,0 +1,435 @@ +# Package Repositories + +Custom FreeBSD, OpenBSD, and Rocky Linux package repository served from k3s, allowing native package tools (`pkg`, `pkg_add`, and `dnf`) to install custom-built packages. + +## Architecture + +- **nginx pod** in k3s `infra` namespace serves static files from a PV +- Path prefixes: `/freebsd/`, `/openbsd/`, and `/rockylinux/` +- Accessible at `https://pkgrepo.f3s.buetow.org` +- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod) +- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile` + +## Required host context + +This reference is package-repo focused, but it still uses a few f3s host names directly: + +- `f0` = FreeBSD host that carries the NFS-backed PV at `/data/nfs/k3svolumes/pkgrepo/` +- `fishfinger` and `blowfish` = OpenBSD frontend hosts used for OpenBSD package install/build tasks +- `r0-r2` = Rocky Linux 9 x86_64 bhyve VMs +- `pi0-pi3` = Rocky Linux 9 aarch64 Raspberry Pi nodes +- `earth` = Fedora laptop used to build, publish, and verify packages + +If you need the broader homelab topology, network roles, or non-package service context, load the sibling `f3s` skill as well. + +## Key Files + +| File | Purpose | +|------|---------| +| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) | +| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest | +| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD | +| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` | +| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends | +| `frontends/Rexfile` (`gogios_install`) | Installs/updates gogios from the custom repo | +| `packages/Makefile` | `make pkg` cross-compiles, packages, and uploads for both OSes; `make dtail-rocky` builds and publishes the Rocky repo | +| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: pkg create + pkg repo + copy to PV | +| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: pkg_create + signify signing | +| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads | + +## PV Directory Structure + +``` +/data/nfs/k3svolumes/pkgrepo/ + freebsd/ + FreeBSD:15:amd64/ + latest/ + All/ # .pkg files + packagesite.pkg # repo metadata (generated by `pkg repo`) + meta / meta.conf + data.pkg + openbsd/ + 7.8/ + packages/ + amd64/ # .tgz files + rockylinux/ + 9/ + x86_64/ # .rpm files + repodata/ + aarch64/ # .rpm files + repodata/ +``` + +Hosted on NFS at f0 (`/data/nfs/k3svolumes/pkgrepo/`). + +## FreeBSD Client Setup (f0–f3) + +Custom repo is pre-configured on f0, f1, f2 (f3 offline at time of setup). +Official FreeBSD repos remain active alongside the custom repo. + +File: `/usr/local/etc/pkg/repos/custom.conf` + +``` +custom: { + url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest", + mirror_type: "NONE", + signature_type: "NONE", + enabled: yes +} +``` + +### Using packages + +```sh +doas pkg update +doas pkg install # works for both official and custom packages +``` + +### Setting up a new FreeBSD host + +Pipe the config via stdin to avoid csh quoting issues (FreeBSD default shell is csh): + +```sh +cat <<'REPO' | ssh -p 22 .lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null' +custom: { + url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest", + mirror_type: "NONE", + signature_type: "NONE", + enabled: yes +} +REPO +``` + +## OpenBSD Client Setup (blowfish/fishfinger) + +Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`. +Official OpenBSD packages still install normally via `/etc/installurl`. + +```sh +export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/" +``` + +### Using packages + +```sh +doas pkg_add # from custom repo (signed with signify) +doas pkg_add # from official repo (still works) +``` + +### Setting up new OpenBSD hosts + +Two things needed: + +1. **Signify public key** — copy from an existing host to `/etc/signify/custom-pkg.pub`: + ```sh + scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/ + scp /tmp/custom-pkg.pub rex@:/tmp/ + ssh rex@ "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub" + ``` + +2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`: + ```sh + rex pkgrepo_setup # adds PKG_PATH to /root/.profile on all frontends + ``` + +The PKG_PATH must be updated when the OpenBSD version changes (currently 7.8). + +## Rocky Linux Client Setup (r0-r2, pi0-pi3) + +The Rocky repo is architecture-specific and follows the standard DNF layout: + +- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/` +- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/` + +### Temporary one-off usage + +```sh +sudo dnf repoquery \ + --disablerepo='*' \ + --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \ + --enablerepo=f3s-dtail \ + dtail +``` + +### Persistent repo file + +Create `/etc/yum.repos.d/f3s-dtail.repo`: + +```ini +[f3s-dtail] +name=f3s DTail +baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/ +enabled=1 +gpgcheck=0 +repo_gpgcheck=0 +``` + +Then: + +```sh +sudo dnf makecache +sudo dnf install dtail +``` + +### Package signing + +OpenBSD packages are signed with `signify(1)` via `pkg_sign`: +- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host) +- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients +- Signing happens automatically during `make pkg-openbsd` / `make pkg` +- `pkg_add` verifies the signature against the public key — no `-D unsigned` needed + +## Packaging Workflow (Makefile) + +Build scripts live in `~/git/conf/packages/`, separate from individual project repos. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV. + +### Build and upload + +```sh +cd ~/git/conf/packages + +# Both FreeBSD and OpenBSD +make pkg NAME=gogios SRC=/home/paul/git/gogios \ + COMMENT="Monitoring tool with email alerts and HTML status page" \ + DESC="Gogios is a lightweight monitoring tool written in Go." + +# Single OS +make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios +make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios + +# DTail Rocky Linux repo (x86_64 + aarch64 RPMs + repodata) +make dtail-rocky +``` + +### How it works + +**Single-binary packages (pure Go, cross-compilable):** +1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`) +2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD) +3. Runs the script via SSH to package natively (`pkg create` / `pkg_create`) +4. OpenBSD packages are signed with signify automatically +5. FreeBSD repo metadata is regenerated with `pkg repo` +6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/` + +**CGo packages (e.g. dtail with DataDog/zstd):** +Cross-compilation from Linux to OpenBSD fails for CGo — needs a C cross-compiler. +Solution: native build on the local OpenBSD VM. Source is sent via `git archive HEAD | ssh ... tar -x` +(not `scp -r`) to avoid filling /tmp with build artifacts and test data. + +**Rocky Linux RPMs (dtail only, current implementation):** +- `x86_64` RPM is built locally on earth +- `aarch64` RPM is packaged on `pi0` because Fedora's x86_64 `rpmbuild` refuses to emit an `aarch64` binary RPM even when the payload is already prebuilt +- repo metadata is generated locally in a `rockylinux:9` container with `createrepo_c` +- the finished repo tree is copied to `/data/nfs/k3svolumes/pkgrepo/rockylinux/9/` + +### Required variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `NAME` | Package name | `gogios` | +| `SRC` | Go project root (must have `cmd//main.go` and `internal/version.go`) | `/home/paul/git/gogios` | + +### Optional variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `COMMENT` | `$(NAME)` | One-line description | +| `DESC` | `$(NAME)` | Longer description | +| `MAINTAINER` | `paul@buetow.org` | Maintainer email | +| `WWW` | `https://buetow.org` | Project URL | +| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC | + +### Version detection + +Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped. + +### Install/update on frontends via Rex + +```sh +cd ~/git/conf/frontends +rex gogios_install # installs or updates gogios on blowfish + fishfinger +rex gogios # full setup (includes gogios_install + config + cron) +``` + +The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD). + +## DTail Package (multi-binary, OpenBSD only) + +DTail is a distributed log tailing tool with 6 binaries, config files, and an rc script. +Unlike single-binary packages, it has a dedicated Makefile target and packaging script. + +### Build and upload + +```sh +cd ~/git/conf/packages +make dtail-openbsd +``` + +### What the package includes + +| File | Purpose | +|------|---------| +| `/usr/local/bin/dserver` | DTail server daemon | +| `/usr/local/bin/dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | Client tools | +| `/etc/dserver/dtail.json` | Server config (from `frontends/etc/dserver/dtail.json.tpl`) | +| `/etc/rc.d/dserver` | RC script (from `frontends/etc/rc.d/dserver.tpl`) | +| `/usr/local/bin/dserver-update-key-cache.sh` | SSH key cache helper for daily cron | + +### Rex integration + +```sh +cd ~/git/conf/frontends +rex dtail_install # install/update dtail package from custom repo +rex dtail # full setup (install + service user + daily cron + start) +``` + +The `dtail` Rex task installs the package, creates the `_dserver` service user, +adds the key cache script to daily cron, and ensures dserver is running. + +## DTail Rocky Linux RPM Repo + +DTail for Rocky is published as a multi-binary RPM for both lab architectures: + +- `dtail-4.3.2-1.ng.x86_64.rpm` +- `dtail-4.3.2-1.ng.aarch64.rpm` + +The RPM currently installs: + +- `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` +- `/usr/local/bin/dserver-update-key-cache.sh` +- `/etc/dserver/dtail.json` +- `/usr/lib/systemd/system/dserver.service` +- `/usr/lib/systemd/system/dserver-update-keycache.service` +- `/usr/lib/systemd/system/dserver-update-keycache.timer` + +### Build and publish + +```sh +cd ~/git/conf/packages +make dtail-rocky +``` + +### Notes + +- The packaged key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys`, so `root` works on `r0-r2` without a separate manual cache copy. +- The packaged `dserver.service` includes the `RuntimeDirectory` and `ExecStartPre` fix required on Rocky so `/var/run/dserver` exists before startup. +- The repo is unsigned for now, so the example repo file uses `gpgcheck=0` and `repo_gpgcheck=0`. + +### Verified DTail package state + +On 2026-04-11 this package repo path was verified with: + +- `dtail-4.3.2-1.ng.x86_64.rpm` and `dtail-4.3.2-1.ng.aarch64.rpm` published at `https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/` +- successful `dnf repoquery` for `dtail` from `r0` (`x86_64`) and `pi0` (`aarch64`) +- successful `dnf install dtail` on `r0-r2` and `pi0-pi3` +- `dserver` active after package install, timer enable, restart, and key-cache refresh on all seven Rocky hosts +- successful `dcat /etc/fstab` reads from earth using `--user root` for `r0-r2` and `--user paul` for `pi0-pi3` + +### OpenBSD DTail package verification + +Also verified on 2026-04-11: + +- `dtail-4.3.2-ng` rebuilt and published to the OpenBSD repo path +- package replacement, key-cache refresh, and successful `dcat /etc/fstab` reads as `--user rex` on `blowfish.buetow.org` and `fishfinger.buetow.org` + +### Why a build VM? + +DTail depends on DataDog/zstd which requires CGo. Cross-compiling CGo from Linux +to OpenBSD needs a C cross-compiler. Instead, a local OpenBSD QEMU/KVM VM builds +natively — this also works for any future packages with C dependencies. + +## OpenBSD Build VM + +A minimal OpenBSD QEMU/KVM VM on the laptop (earth) for native compilation. +Scripts in `packages/buildvm/`. + +### Initial setup (once) + +```sh +cd ~/git/conf/packages/buildvm + +# Fetch signify keys from fishfinger +scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec . +scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub . + +./setup.sh # downloads ISO, runs fully automated install (~5 min) +./provision.sh # installs Go, git, gmake, signify keys, SSH keys +``` + +`setup.sh` is fully automated — it calls `install-expect.exp` which drives the +OpenBSD serial console installer via expect. No manual interaction needed. + +### Day-to-day use + +```sh +make buildvm-start # boot VM (~15s) +make dtail-openbsd # auto-starts VM if needed +make buildvm-stop # shut down when done +``` + +The VM is headless, SSH on `localhost:2222`, 1GB RAM, 2 CPUs, 4GB disk. +Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`. + +### Automated installer notes (install-expect.exp) + +- The expect script is in a **separate file** (`install-expect.exp`) to avoid bash/expect quoting interactions — embedding expect in a bash heredoc causes password send failures +- Serial console is activated via `set tty com0` at the OpenBSD boot prompt +- Password prompts require `sleep 2` before `send` — `sleep 1` is not enough for the serial console to be ready +- OpenBSD 7.8 added a new "Encrypt the root disk?" prompt before the partition layout +- After CONGRATULATIONS, choose shell (`s`) instead of reboot — the CD is still attached and reboot re-boots the installer; configure wheel group via `chroot /mnt` +- Username `build` is rejected by the installer ("not a usable loginname") — use `pbuild` instead +- Source code is synced to the VM via `git archive HEAD | ssh ... tar -x` to avoid filling /tmp (full repo with benchmarks exceeds available space on 4GB disk) + +## Building Custom Packages + +### FreeBSD + +On any FreeBSD host, or via the Mage SCP-and-remote-execute pattern: + +```sh +pkg create -M +MANIFEST -p plist -r stagedir -o output/All +pkg repo output/ # regenerates repo metadata (unsigned) +``` + +Copy output to PV: +```sh +doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/ +``` + +### OpenBSD + +On an OpenBSD host: + +```sh +pkg_create \ + -D COMMENT="Package description" \ + -d descfile \ + -f packing-list \ + -B stagedir \ + -p / \ + output/package-name-1.0.tgz +``` + +Copy to PV via f0: +```sh +scp package.tgz f0.lan.buetow.org:/tmp/ +ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/" +``` + +## SSH Access for Package Tasks + +| Host | Access | Root | Notes | +|------|--------|------|-------| +| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local, default shell is csh | +| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | Packages built here, then copied to f0 PV | +| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger | + +## Notes + +- HTTP is always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://` +- OpenBSD version in the path must match the host OS version (currently 7.8) +- FreeBSD version follows the ABI naming convention (`FreeBSD:15:amd64`) +- OpenBSD packages are signed with signify (key pair at `/etc/signify/custom-pkg.*`) +- FreeBSD packages are unsigned (`signature_type: "NONE"`) — signing with RSA is possible but not yet configured +- After adding/removing FreeBSD packages, always regenerate repo metadata with `pkg repo` +- OpenBSD does not need a repo index — `pkg_add` fetches packages directly by name +- FreeBSD hosts use csh as default shell — remote scripts must use `/bin/sh` explicitly +- f3 is currently offline; configure its custom repo when it comes back online -- cgit v1.2.3