skills: split package repo docs out of f3s

4 files changed, 220 insertions, 18 deletions

diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md
index 5757862..9aabebc 100644
--- a/prompts/skills/f3s/SKILL.md
+++ b/prompts/skills/f3s/SKILL.md
@@ -25,12 +25,13 @@ Detailed reference documentation is in the `references/` subfolder:
 - [Storage](references/storage.md) — ZFS (zdata), CARP, NFS over stunnel, zrepl replication
 - [k3s Setup](references/k3s-setup.md) — HA k3s cluster, etcd, node IPs, kubeconfig, ArgoCD
 - [Observability](references/observability.md) — Prometheus, Grafana, Loki, Alloy, Tempo
-- [Package Repos](references/package-repos.md) — Custom FreeBSD/OpenBSD pkg repo served from k3s nginx pod
 - [Immich](references/immich.md) — Photo server deployment, job queue stats, troubleshooting
 - [Garage](references/garage.md) — Garage cluster, edge domain routing, S3 bucket/key workflow, troubleshooting
 - [DTail / dserver](references/dtail.md) — dserver: Pis **arm64** vs r0–r2 **amd64**, r-VM **root** + `root.authorized_keys` cache, firewalld **2222**, systemd timers
 - [dserver.d](references/dserver.d) — index: links to **Rocky r-VM DTail** subsection and full **dtail.md**
 
+Package repository details were split into the sibling `pkgrepo` skill. Use `pkgrepo` for `pkgrepo.f3s.buetow.org`, repo layout, package publication, and client repo configuration.
+
 ## Quick Reference: Host IPs
 
 | Host | Role | LAN IP | WireGuard IP |
diff --git a/prompts/skills/f3s/references/dtail.md b/prompts/skills/f3s/references/dtail.md
index bfee00a..5d6a2a8 100644
--- a/prompts/skills/f3s/references/dtail.md
+++ b/prompts/skills/f3s/references/dtail.md
@@ -10,8 +10,15 @@ Upstream install and examples live in the repo: `doc/installation.md`, `examples
 |-------|-----------|----------------|------------------|
 | **pi0–pi3** | Rocky Linux 9 **aarch64** (Raspberry Pi 3) | Cross-build **linux/arm64**, `nozstd` | `paul@piN.lan.buetow.org` |
 | **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` |
+| **blowfish, fishfinger** | OpenBSD 7.8 **amd64** | Native OpenBSD package build | `rex@blowfish.buetow.org`, `rex@fishfinger.buetow.org` |
 
-**Root login and key cache:** `examples/update_key_cache.sh.example` only scans `/home/*`. If clients connect as **root**, copy keys once (e.g. after install) and on key changes:
+**Key cache filenames matter:** `examples/update_key_cache.sh.example` only scans `/home/*` and writes `/var/run/dserver/cache/USER.authorized_keys`. In this lab, DTail auth worked only after writing the exact cache filename for the login user:
+
+- **r0–r2**: `root.authorized_keys`
+- **pi0–pi3**: `paul.authorized_keys`
+- **blowfish, fishfinger**: `rex.authorized_keys`
+
+If clients connect as **root**, copy keys once (e.g. after install) and on key changes:
 
 ```bash
 cp /root/.ssh/authorized_keys /var/run/dserver/cache/root.authorized_keys
@@ -19,6 +26,14 @@ chown dserver:dserver /var/run/dserver/cache/root.authorized_keys
 chmod 600 /var/run/dserver/cache/root.authorized_keys
 ```
 
+For the Pi nodes:
+
+```bash
+cp /home/paul/.ssh/authorized_keys /var/run/dserver/cache/paul.authorized_keys
+chown dserver:dserver /var/run/dserver/cache/paul.authorized_keys
+chmod 600 /var/run/dserver/cache/paul.authorized_keys
+```
+
 ## dserver on r0, r1, r2 (k3s Rocky VMs, amd64)
 
 These hosts are the **x86_64** guests on f0/f1/f2. SSH and VM background: [Rocky Linux VMs](rocky-linux-vms.md), **DTail subsection** (same content in short form): [DTail (dserver) on r0–r2](rocky-linux-vms.md#dtail-dserver-on-r0r2). Shortcut index file: [dserver.d](dserver.d). **Do not** install the Pi **arm64** binary here.
@@ -30,14 +45,19 @@ These hosts are the **x86_64** guests on f0/f1/f2. SSH and VM background: [Rocky
 | Admin SSH (normal) | `ssh -p 22 root@rN.lan.buetow.org` (key-based; see Rocky VM doc) |
 | dserver port | **2222/tcp** (DTail clients); **22** remains sshd |
 
-### Build the binary (on earth)
+### Build the binaries (on earth)
 
 ```bash
 cd ~/git/dtail   # your checkout of https://codeberg.org/snonux/dtail
-CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver
-# artifact: ./dserver  (static amd64; copy as e.g. dserver-linux-amd64)
+
+# exact cross-builds for the Rocky VMs
+for bin in dserver dtail dcat dgrep dmap dtailhealth; do
+  CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o "$bin-linux-amd64" ./cmd/$bin/main.go
+done
 ```
 
+Use direct `go build -tags nozstd` when you need a deterministic cross-build artifact. Reusing `make dserver` after a local native build can leave an old binary in place because the target name is just `dserver`.
+
 ### Install on each rN (run as root over SSH)
 
 1. Copy to the VM, e.g. `scp -P 22 dserver root@r0.lan.buetow.org:/tmp/` and the files from `examples/` (see checklist below), or unpack a small staging dir on `/tmp/dtail-install/`.
@@ -65,6 +85,22 @@ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver
    systemctl start dserver-update-keycache.service    # populate cache once
    ```
 
+   **Important:** the stock unit uses `WorkingDirectory=/var/run/dserver`. On Rocky and Pi hosts, `dserver` failed with `status=200/CHDIR` until the unit also recreated that tmpfs path at service start. Add:
+
+   ```ini
+   RuntimeDirectory=dserver
+   RuntimeDirectoryMode=0755
+   ExecStartPre=/usr/bin/mkdir -p /var/run/dserver/cache /var/run/dserver/log
+   ```
+
+   Then run:
+
+   ```bash
+   systemctl daemon-reload
+   systemctl reset-failed dserver
+   systemctl start dserver
+   ```
+
 5. **Root SSH key cache (required on r VMs)**  
    The stock `update_key_cache.sh` only copies `/home/*/.ssh/authorized_keys`. **Root’s keys live in `/root`**, so mirror them explicitly (after any change to root’s authorized_keys):
 
@@ -95,9 +131,9 @@ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver
 DTail uses the **same username** as normal SSH unless overridden. If dserver is set up for **root** only on r VMs, run as root or pass the client user your config expects. Example:
 
 ```bash
-dcat --servers r0.lan,r1.lan,r2.lan /etc/hostname
-# if your client user must be root for key + dtail.json to match:
-# sudo dcat ...   # or configure SSH user flags if the client supports them
+dcat --plain --noColor --trustAllHosts --user root \
+  --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org \
+  --files /etc/fstab
 ```
 
 Add **2222** host keys to `~/.ssh/known_hosts` the first time (interactive trust, or `ssh-keyscan -p 2222`).
@@ -110,12 +146,10 @@ From a clone of the repo:
 cd ~/git/dtail   # or your checkout
 
 # Raspberry Pi 4× — linux/arm64, static, no CGO zstd
-CGO_ENABLED=0 GOOS=linux GOARCH=arm64 DTAIL_NO_ZSTD=yes make dserver
-# produces ./dserver — keep a copy e.g. ./dserver-linux-arm64
+CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -tags nozstd -o dserver-linux-arm64 ./cmd/dserver/main.go
 
 # k3s VMs r0–r2 — linux/amd64
-CGO_ENABLED=0 GOOS=linux GOARCH=amd64 DTAIL_NO_ZSTD=yes make dserver
-# produces ./dserver — keep a copy e.g. ./dserver-linux-amd64
+CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o dserver-linux-amd64 ./cmd/dserver/main.go
 ```
 
 `DTAIL_NO_ZSTD=yes` sets the `nozstd` build tag so the binary does not link DataDog’s CGO zstd (required for static cross-compiles). **`.zst` log files are not supported** in that binary; gzip still works.
@@ -128,7 +162,7 @@ Do this on **each** target (pi0–pi3 and/or r0–r2). Adjust **user** if you ar
 2. **OS user**: `dserver` system account (`useradd -r -d /var/lib/dserver -s /sbin/nologin -U dserver`).
 3. **Directories**: `/etc/dserver`, `/var/run/dserver` (tmpfs — recreated at boot; use systemd `RuntimeDirectory` in the example unit if you adopt it).
 4. **Config**: `/etc/dserver/dtail.json` from `examples/dtail.json.example` — ensure **Server.Permissions.Users** includes every login name that will connect (e.g. `paul`, `root`).
-5. **systemd**: `examples/dserver.service.example` → `/etc/systemd/system/dserver.service` — unit stays **disabled** by default; start with `systemctl start dserver`.
+5. **systemd**: `examples/dserver.service.example` → `/etc/systemd/system/dserver.service` — unit stays **disabled** by default; start with `systemctl start dserver`. Add `RuntimeDirectory=dserver`, `RuntimeDirectoryMode=0755`, and `ExecStartPre=/usr/bin/mkdir -p /var/run/dserver/cache /var/run/dserver/log` so the tmpfs working directory exists before `dserver` starts.
 6. **SSH host keys for clients**: dserver cannot read users’ `~/.ssh/authorized_keys` as user `dserver`. Use `examples/update_key_cache.sh.example` + `dserver-update-keycache.service` / `.timer` to mirror keys into `/var/run/dserver/cache/USER.authorized_keys`.
 7. **firewalld**: allow **2222/tcp** (ping may work while TCP is blocked):
 
@@ -149,12 +183,39 @@ Do this on **each** target (pi0–pi3 and/or r0–r2). Adjust **user** if you ar
 ## Client examples
 
 ```bash
-dcat --servers pi0.lan,pi1.lan,pi2.lan,pi3.lan /etc/os-release
-dcat --servers r0.lan,r1.lan,r2.lan /etc/hostname
+dcat --plain --noColor --trustAllHosts --user paul \
+  --servers pi0.lan.buetow.org,pi1.lan.buetow.org,pi2.lan.buetow.org,pi3.lan.buetow.org \
+  --files /etc/fstab
+
+dcat --plain --noColor --trustAllHosts --user root \
+  --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org \
+  --files /etc/fstab
 ```
 
 Use hostnames that resolve from where you run the client (often `*.lan.buetow.org`).
 
+## Package-managed DTail
+
+For package-repo-backed DTail and other custom package repo tasks, use the sibling `pkgrepo` skill and its reference:
+
+- [Package Repositories](../../pkgrepo/references/package-repos.md)
+
+That skill now owns:
+
+- FreeBSD, OpenBSD, and Rocky client repo configuration
+- custom repo layout, publication, and verification notes
+- Rocky repo client configuration and RPM install flow
+- OpenBSD DTail package build, publish, replace, and cache-refresh steps
+- DTail package publication details that depend on the repo
+
+## Verified lab state
+
+On 2026-04-11 this setup was verified end-to-end with:
+
+- the full Linux binary set (`dserver`, `dtail`, `dcat`, `dgrep`, `dmap`, `dtailhealth`) installed on `r0`-`r2` and `pi0`-`pi3`
+- `dserver` active and listening on `*:2222` on all seven hosts
+- successful `dcat /etc/fstab` reads from earth using `--user root` for `r0`-`r2` and `--user paul` for `pi0`-`pi3`
+
 ## k3s / r0–r2 notes
 
 - **amd64** binaries only; do not deploy the Pi **arm64** build there.
diff --git a/prompts/skills/pkgrepo/SKILL.md b/prompts/skills/pkgrepo/SKILL.md
new file mode 100644
index 0000000..77de23e
--- /dev/null
+++ b/prompts/skills/pkgrepo/SKILL.md
@@ -0,0 +1,31 @@
+---
+name: pkgrepo
+description: Reference skill for the homelab package repositories behind pkgrepo.f3s.buetow.org. Covers FreeBSD pkg, OpenBSD pkg_add, Rocky Linux dnf repos, PV layout, nginx/ArgoCD wiring, DTail package publishing, and client repo configuration. Use when publishing packages, troubleshooting repository access or metadata, or deciding how packages should be installed from the custom repo.
+---
+
+# Package Repo Reference
+
+Use this skill when the task is specifically about the custom package repositories rather than the broader f3s homelab.
+
+## When to Use
+
+- Publishing or updating packages in `pkgrepo.f3s.buetow.org`
+- Troubleshooting repo layout, metadata, or HTTP exposure
+- Configuring FreeBSD, OpenBSD, or Rocky Linux clients to install from the custom repo
+- Working on DTail package publishing and installation through the repo
+
+## Reference Files
+
+- [Package Repositories](references/package-repos.md) — repo layout, PV paths, nginx exposure, client config, and DTail package workflows
+
+## Scope
+
+This skill owns package repository details that used to live under `f3s`.
+
+Use `f3s` alongside this skill when the task depends on broader host-role or cluster context, especially:
+
+- `f0` as the FreeBSD NFS/PV host for `/data/nfs/k3svolumes/pkgrepo/`
+- `fishfinger` and `blowfish` as the OpenBSD frontend hosts
+- `r0-r2` as Rocky Linux x86_64 bhyve VMs
+- `pi0-pi3` as Rocky Linux aarch64 Raspberry Pi nodes
+- `earth` as the Fedora laptop used for package publication and verification
diff --git a/prompts/skills/f3s/references/package-repos.md b/prompts/skills/pkgrepo/references/package-repos.md
index f8010be..d6c45c4 100644
--- a/prompts/skills/f3s/references/package-repos.md
+++ b/prompts/skills/pkgrepo/references/package-repos.md
@@ -1,15 +1,27 @@
 # Package Repositories
 
-Custom FreeBSD and OpenBSD package repository served from k3s, allowing native package tools (`pkg` and `pkg_add`) to install custom-built packages.
+Custom FreeBSD, OpenBSD, and Rocky Linux package repository served from k3s, allowing native package tools (`pkg`, `pkg_add`, and `dnf`) to install custom-built packages.
 
 ## Architecture
 
 - **nginx pod** in k3s `infra` namespace serves static files from a PV
-- Path prefixes: `/freebsd/` and `/openbsd/`
+- Path prefixes: `/freebsd/`, `/openbsd/`, and `/rockylinux/`
 - Accessible at `https://pkgrepo.f3s.buetow.org`
 - TLS terminated by OpenBSD relayd on the internet gateways (not in the pod)
 - DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile`
 
+## Required host context
+
+This reference is package-repo focused, but it still uses a few f3s host names directly:
+
+- `f0` = FreeBSD host that carries the NFS-backed PV at `/data/nfs/k3svolumes/pkgrepo/`
+- `fishfinger` and `blowfish` = OpenBSD frontend hosts used for OpenBSD package install/build tasks
+- `r0-r2` = Rocky Linux 9 x86_64 bhyve VMs
+- `pi0-pi3` = Rocky Linux 9 aarch64 Raspberry Pi nodes
+- `earth` = Fedora laptop used to build, publish, and verify packages
+
+If you need the broader homelab topology, network roles, or non-package service context, load the sibling `f3s` skill as well.
+
 ## Key Files
 
 | File | Purpose |
@@ -20,9 +32,10 @@ Custom FreeBSD and OpenBSD package repository served from k3s, allowing native p
 | `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` |
 | `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends |
 | `frontends/Rexfile` (`gogios_install`) | Installs/updates gogios from the custom repo |
-| `packages/Makefile` | `make pkg` cross-compiles, packages, and uploads for both OSes |
+| `packages/Makefile` | `make pkg` cross-compiles, packages, and uploads for both OSes; `make dtail-rocky` builds and publishes the Rocky repo |
 | `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: pkg create + pkg repo + copy to PV |
 | `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: pkg_create + signify signing |
+| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads |
 
 ## PV Directory Structure
 
@@ -39,6 +52,10 @@ Custom FreeBSD and OpenBSD package repository served from k3s, allowing native p
     7.8/
       packages/
         amd64/            # .tgz files
+  rockylinux/
+    9/
+      x86_64/             # .rpm files + repodata/
+      aarch64/            # .rpm files + repodata/
 ```
 
 Hosted on NFS at f0 (`/data/nfs/k3svolumes/pkgrepo/`).
@@ -115,6 +132,43 @@ Two things needed:
 
 The PKG_PATH must be updated when the OpenBSD version changes (currently 7.8).
 
+## Rocky Linux Client Setup (r0-r2, pi0-pi3)
+
+The Rocky repo is architecture-specific and follows the standard DNF layout:
+
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/`
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/`
+
+### Temporary one-off usage
+
+```sh
+sudo dnf repoquery \
+  --disablerepo='*' \
+  --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \
+  --enablerepo=f3s-dtail \
+  dtail
+```
+
+### Persistent repo file
+
+Create `/etc/yum.repos.d/f3s-dtail.repo`:
+
+```ini
+[f3s-dtail]
+name=f3s DTail
+baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/
+enabled=1
+gpgcheck=0
+repo_gpgcheck=0
+```
+
+Then:
+
+```sh
+sudo dnf makecache
+sudo dnf install dtail
+```
+
 ### Package signing
 
 OpenBSD packages are signed with `signify(1)` via `pkg_sign`:
@@ -140,6 +194,9 @@ make pkg NAME=gogios SRC=/home/paul/git/gogios \
 # Single OS
 make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios
 make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios
+
+# DTail Rocky Linux repo (x86_64 + aarch64 RPMs + repodata)
+make dtail-rocky
 ```
 
 ### How it works
@@ -157,6 +214,12 @@ Cross-compilation from Linux to OpenBSD fails for CGo — needs a C cross-compil
 Solution: native build on the local OpenBSD VM. Source is sent via `git archive HEAD | ssh ... tar -x`
 (not `scp -r`) to avoid filling /tmp with build artifacts and test data.
 
+**Rocky Linux RPMs (dtail only, current implementation):**
+- `x86_64` RPM is built locally on earth
+- `aarch64` RPM is packaged on `pi0` because Fedora's x86_64 `rpmbuild` refuses to emit an `aarch64` binary RPM even when the payload is already prebuilt
+- repo metadata is generated locally in a `rockylinux:9` container with `createrepo_c`
+- the finished repo tree is copied to `/data/nfs/k3svolumes/pkgrepo/rockylinux/9/`
+
 ### Required variables
 
 | Variable | Description | Example |
@@ -221,6 +284,52 @@ rex dtail           # full setup (install + service user + daily cron + start)
 The `dtail` Rex task installs the package, creates the `_dserver` service user,
 adds the key cache script to daily cron, and ensures dserver is running.
 
+## DTail Rocky Linux RPM Repo
+
+DTail for Rocky is published as a multi-binary RPM for both lab architectures:
+
+- `dtail-4.3.2-1.ng.x86_64.rpm`
+- `dtail-4.3.2-1.ng.aarch64.rpm`
+
+The RPM currently installs:
+
+- `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth`
+- `/usr/local/bin/dserver-update-key-cache.sh`
+- `/etc/dserver/dtail.json`
+- `/usr/lib/systemd/system/dserver.service`
+- `/usr/lib/systemd/system/dserver-update-keycache.service`
+- `/usr/lib/systemd/system/dserver-update-keycache.timer`
+
+### Build and publish
+
+```sh
+cd ~/git/conf/packages
+make dtail-rocky
+```
+
+### Notes
+
+- The packaged key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys`, so `root` works on `r0-r2` without a separate manual cache copy.
+- The packaged `dserver.service` includes the `RuntimeDirectory` and `ExecStartPre` fix required on Rocky so `/var/run/dserver` exists before startup.
+- The repo is unsigned for now, so the example repo file uses `gpgcheck=0` and `repo_gpgcheck=0`.
+
+### Verified DTail package state
+
+On 2026-04-11 this package repo path was verified with:
+
+- `dtail-4.3.2-1.ng.x86_64.rpm` and `dtail-4.3.2-1.ng.aarch64.rpm` published at `https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/`
+- successful `dnf repoquery` for `dtail` from `r0` (`x86_64`) and `pi0` (`aarch64`)
+- successful `dnf install dtail` on `r0-r2` and `pi0-pi3`
+- `dserver` active after package install, timer enable, restart, and key-cache refresh on all seven Rocky hosts
+- successful `dcat /etc/fstab` reads from earth using `--user root` for `r0-r2` and `--user paul` for `pi0-pi3`
+
+### OpenBSD DTail package verification
+
+Also verified on 2026-04-11:
+
+- `dtail-4.3.2-ng` rebuilt and published to the OpenBSD repo path
+- package replacement, key-cache refresh, and successful `dcat /etc/fstab` reads as `--user rex` on `blowfish.buetow.org` and `fishfinger.buetow.org`
+
 ### Why a build VM?
 
 DTail depends on DataDog/zstd which requires CGo. Cross-compiling CGo from Linux