Run a dev stack beside production, and write both flows down #72

Merged

Grandiras merged 2 commits from issue-69-dev-environment into main 2026-08-08 01:53:51 +00:00

Conversation 2 Commits 2 Files changed 15 +776 -73

Grandiras commented 2026-08-08 01:50:27 +00:00

Owner

Copy link

Fixes #69.

dev.ponente.app is a second stack on the production server — its own app, PDS, database, volumes and encryption key — running whatever is on main. It is already up and verified; this PR is the tooling and the write-up behind it.

Verified on the server

check	result
https://dev.ponente.app/healthz	200, cert CN=dev.ponente.app issued by Let's Encrypt on first handshake
https://pds.dev.ponente.app/xrpc/_health	200
production containers	untouched — still v2.3.0, Up 10 days
admin seeding	both maintainer DIDs present, Role=1
mail	startup log reports it configured (off in the local stack)
ask endpoint	dev site → 200, junk → 403, alice.dev.ponente.app → 404 at the dev PDS (0 hits on production's)
auto-update timer	active, 60 s cadence, runs succeed as no-ops in 459 ms
dotnet test	153/153

The handle check is the one that matters: a missing Host rewrite 404s identically, so this was confirmed from the dev PDS's own request log rather than the status code.

Nobody deploys dev

Merging a PR publishes ponente-kanban:main; a systemd user timer on the host recreates the dev app container within a minute, --no-deps so the database and PDS keep running.

CI cannot push that update, and this was measured rather than assumed. Jobs run inside the dind daemon, and a container on the host's rootless podman cannot open a connection to the host's own public address on port 22 — 443 through Caddy works, which is exactly why it looks like SSH should. The mechanism that can reach in is mounting the podman socket into a Watchtower-style container, which hands that container every container on the host, production included. A minute of latency is the better trade, so the direction is inverted and the host asks the registry.

update.sh refuses any IMAGE_TAG matching v[0-9], so the timer can never move production.

One compose file, not a fork

STACK_NAME names the compose project and therefore the containers, network and volumes. podman-compose config against the production env renders byte-identical to before, plus an empty Allowlist__Admins__1 that SeedAdminsAsync already drops. A production redeploy from this branch is a no-op change.

Host-side changes, outside the repo

/home/podman/caddy/ask.Caddyfile gained the three dev site names and a .dev.ponente.app handler above the .ponente.app one — the deeper suffix matches both endsWith tests and handle blocks run in written order, so production's PDS would otherwise be asked about a dev handle and correctly say no. Validated with caddy validate before installing, and the previous file is at ask.Caddyfile.bak-pre-dev. Caddy picked it up on the dev deploy's container events; no restart, which would have dropped every other project's sites.

Nothing else on the host changed. No DNS was needed — the registrar's *.ponente.app wildcard answers at any depth.

Documentation

deploy/README.md is the runbook: both flows, first-time setup, the ask.Caddyfile content, and the three container naming schemes. README.md and CLAUDE.md point at it.

The release skill's Phase A no longer builds a stack for hand-testing — dev has held main since each merge, so Phase A confirms rather than produces. That also retires the "don't run the e2e suite while testing by hand" warning: the suite runs against the local stack and can no longer collide with anyone.

Two smaller fixes found on the way

• check-env-keys.sh matched keys with [A-Z_]+, so ADMIN_DID_2 and any other name containing a digit were silently skipped — a key missing everywhere and reported nowhere. It also takes --env dev now.
• .gitignore matched .env exactly, leaving deploy/.env.dev and the deploy/.env.bak somebody makes before editing a secret untracked but not ignored, one git add -A from being committed.

Worth knowing

Mail from dev is real mail and accounts minted on its PDS are real Atmosphere identities on plc.directory. That is the point — those paths reached production unexercised before — but it is stated in both deploy/README.md and the release skill so it is never a surprise.

🤖 Generated with Claude Code

Fixes #69. `dev.ponente.app` is a second stack on the production server — its own app, PDS, database, volumes and encryption key — running whatever is on `main`. **It is already up and verified**; this PR is the tooling and the write-up behind it. ## Verified on the server | check | result | |---|---| | `https://dev.ponente.app/healthz` | 200, cert `CN=dev.ponente.app` issued by Let's Encrypt on first handshake | | `https://pds.dev.ponente.app/xrpc/_health` | 200 | | production containers | untouched — still `v2.3.0`, `Up 10 days` | | admin seeding | both maintainer DIDs present, `Role=1` | | mail | startup log reports it **configured** (off in the local stack) | | ask endpoint | dev site → 200, junk → 403, `alice.dev.ponente.app` → 404 **at the dev PDS** (0 hits on production's) | | auto-update timer | active, 60 s cadence, runs succeed as no-ops in 459 ms | | `dotnet test` | 153/153 | The handle check is the one that matters: a missing `Host` rewrite 404s identically, so this was confirmed from the dev PDS's own request log rather than the status code. ## Nobody deploys dev Merging a PR publishes `ponente-kanban:main`; a systemd user timer on the host recreates the dev app container within a minute, `--no-deps` so the database and PDS keep running. **CI cannot push that update, and this was measured rather than assumed.** Jobs run inside the dind daemon, and a container on the host's rootless podman cannot open a connection to the host's own public address on port 22 — 443 through Caddy works, which is exactly why it looks like SSH should. The mechanism that *can* reach in is mounting the podman socket into a Watchtower-style container, which hands that container every container on the host, production included. A minute of latency is the better trade, so the direction is inverted and the host asks the registry. `update.sh` refuses any `IMAGE_TAG` matching `v[0-9]`, so the timer can never move production. ## One compose file, not a fork `STACK_NAME` names the compose project and therefore the containers, network and volumes. `podman-compose config` against the production env renders byte-identical to before, plus an empty `Allowlist__Admins__1` that `SeedAdminsAsync` already drops. **A production redeploy from this branch is a no-op change.** ## Host-side changes, outside the repo `/home/podman/caddy/ask.Caddyfile` gained the three dev site names and a `.dev.ponente.app` handler **above** the `.ponente.app` one — the deeper suffix matches both `endsWith` tests and `handle` blocks run in written order, so production's PDS would otherwise be asked about a dev handle and correctly say no. Validated with `caddy validate` before installing, and the previous file is at `ask.Caddyfile.bak-pre-dev`. Caddy picked it up on the dev deploy's container events; no restart, which would have dropped every other project's sites. Nothing else on the host changed. No DNS was needed — the registrar's `*.ponente.app` wildcard answers at any depth. ## Documentation `deploy/README.md` is the runbook: both flows, first-time setup, the `ask.Caddyfile` content, and the three container naming schemes. `README.md` and `CLAUDE.md` point at it. The release skill's **Phase A no longer builds a stack for hand-testing** — dev has held `main` since each merge, so Phase A confirms rather than produces. That also retires the "don't run the e2e suite while testing by hand" warning: the suite runs against the local stack and can no longer collide with anyone. ## Two smaller fixes found on the way - `check-env-keys.sh` matched keys with `[A-Z_]+`, so `ADMIN_DID_2` and any other name containing a digit were silently skipped — a key missing everywhere and reported nowhere. It also takes `--env dev` now. - `.gitignore` matched `.env` exactly, leaving `deploy/.env.dev` and the `deploy/.env.bak` somebody makes before editing a secret untracked but *not* ignored, one `git add -A` from being committed. ## Worth knowing Mail from dev is **real** mail and accounts minted on its PDS are **real** Atmosphere identities on `plc.directory`. That is the point — those paths reached production unexercised before — but it is stated in both `deploy/README.md` and the release skill so it is never a surprise. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Grandiras added 1 commit 2026-08-08 01:50:27 +00:00

Run a dev stack beside production, and write both flows down ... 5894594d30

`dev.ponente.app` is a second stack on the production server — its own app,
PDS, database, volumes and encryption key — running whatever is on `main`.
It is already up and serving.

## Nobody deploys it

Merging a PR publishes `ponente-kanban:main`, and a systemd user timer on the
host recreates the dev app container within a minute. `--no-deps`, so the
database and the PDS keep running: they are not what changed, and restarting
the PDS for an app build drops its federation connections for no reason.

**CI cannot push that update, and this was measured rather than assumed.** Jobs
run inside the dind daemon, and a container on the host's rootless podman
cannot open a connection to the host's own public address on port 22 — 443
through Caddy works, which is exactly why it looks like SSH should. The
mechanism that *can* reach in is mounting the podman socket into a
Watchtower-style container, which hands that container every container on the
host, production included. A minute of latency is the better trade, so the
direction is inverted and the host asks the registry. A no-op tick costs 459 ms.

`update.sh` refuses any `IMAGE_TAG` matching `v[0-9]`: a timer that could move
production is a way to ship an untested `main` to the whole group because
somebody edited the wrong env file.

## One compose file, not a fork

`STACK_NAME` names the compose project and therefore the containers, the
network and the volumes, so `docker-compose.prod.yml` describes production with
`deploy/.env` and dev with `deploy/.env.dev`. A dev environment is only worth
having if it is the same shape as production, and a second file drifts from
this one the first time a service grows a setting. Everything that legitimately
differs is already a variable. `podman-compose config` against the production
env renders byte-identical to before, plus an empty `Allowlist__Admins__1` that
`SeedAdminsAsync` already drops.

`ADMIN_DID_2` seeds a second bootstrap admin — both maintainer handles on dev,
empty in production.

## What dev buys, and what it costs

Mail, real certificates, on-demand TLS and a federating PDS are all exercised
there. Mail was the standing example of a path that reached production
unexercised, because the local stack has no mail server and the e2e suite must
not send any. The flip side, stated plainly in the docs: mail from dev is real
mail, and accounts minted on its PDS are real Atmosphere identities.

## Documentation, which is half the issue

`deploy/README.md` is the runbook — both flows, first-time setup, the host-side
`ask.Caddyfile` changes (the `.dev.ponente.app` handler must sit *above* the
`.ponente.app` one; the deeper suffix matches both and `handle` blocks run in
written order), and the three container naming schemes that are easy to
confuse. `README.md` and `CLAUDE.md` point at it.

The release skill's Phase A no longer builds a stack for hand-testing: dev has
held `main` since each merge, so Phase A confirms rather than produces. That
also retires the "don't run the e2e suite while testing by hand" warning — the
suite runs against the local stack and can no longer collide with anyone.

## Two smaller fixes found on the way

- `check-env-keys.sh` matched keys with `[A-Z_]+`, so `ADMIN_DID_2` and any
  other name containing a digit were silently skipped — a key missing
  everywhere and reported nowhere. It also takes `--env dev` now.
- `.gitignore` matched `.env` exactly, leaving `deploy/.env.dev` and the
  `deploy/.env.bak` somebody makes before editing a secret untracked but *not*
  ignored, one `git add -A` from being committed.

Fixes #69

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

claude-bot commented 2026-08-08 01:50:30 +00:00

Collaborator

Copy link

🤖 Reviewing this PR — follow along in the action run; the reply will link the full session transcript.

📜 Live transcript — updating every ~5 minutes while the task runs.

🤖 Reviewing this PR — follow along in [the action run](https://git.grandiras.net/Grandiras/claude-bot/actions/runs/175); the reply will link the full session transcript. 📜 [Live transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/Ponente/pr-72/run-175.md) — updating every ~5 minutes while the task runs.

Grandiras added 1 commit 2026-08-08 01:51:56 +00:00

Spell the health wait as an if, not a trailing && ... 2b782dde3c

Same reasoning the Publish workflow now carries: under `set -e` a
trailing test that is simply false is one edit away from reading as a
failed script. It was safe here — the test is not the last command in
its list — but "safe because of an exemption in the set -e rules" is not
what the rest of this file is written to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

claude-reviewer approved these changes 2026-08-08 01:53:29 +00:00

claude-reviewer left a comment

Copy link

✅ Verdict: approve — no blocking issues found.

This is a thorough, well-reasoned change. I traced the argument parsing and env-file handling in deploy.sh/check-env-keys.sh/update.sh end to end, checked the .gitignore negation patterns against the specific files they're meant to catch (.env, .env.dev, .env.bak, .env.dev.example), and verified SeedAdminsAsync really does drop blank/non-did: entries the way ADMIN_DID_2's comment and deploy/docker-compose.prod.yml assume — all consistent with the claims in the PR description. The STACK_NAME/container_name parameterization in the compose file is a straightforward variable substitution of an already-working mechanism, so it shouldn't regress production. The systemd auto-update timer, the CI publish-on-main change (with the bash -e-safe if instead of &&), and the ask.Caddyfile ordering rationale all check out.

One thing not covered by anything in the diff or the verification table: whether the server user has loginctl enable-linger set, which is what lets a systemd --user timer keep running without an active login session. Given the PR states the timer is already "active, 60 s cadence" on the live server, this is presumably already handled outside the repo — worth a one-line mention in deploy/README.md's troubleshooting section for the next person who tries to reproduce this setup on a different host, but not a blocker.

No correctness or security issues found in the shell scripts, workflow YAML, or compose changes.

---

⏱ 2m 46s · 17 turns · tokens: 90.3k in / 13.0k out (+1716.2k cache-read)
📜 Full session transcript

✅ **Verdict: approve** — no blocking issues found. This is a thorough, well-reasoned change. I traced the argument parsing and env-file handling in `deploy.sh`/`check-env-keys.sh`/`update.sh` end to end, checked the `.gitignore` negation patterns against the specific files they're meant to catch (`.env`, `.env.dev`, `.env.bak`, `.env.dev.example`), and verified `SeedAdminsAsync` really does drop blank/non-`did:` entries the way `ADMIN_DID_2`'s comment and `deploy/docker-compose.prod.yml` assume — all consistent with the claims in the PR description. The `STACK_NAME`/`container_name` parameterization in the compose file is a straightforward variable substitution of an already-working mechanism, so it shouldn't regress production. The systemd auto-update timer, the CI publish-on-main change (with the bash -e-safe `if` instead of `&&`), and the `ask.Caddyfile` ordering rationale all check out. One thing not covered by anything in the diff or the verification table: whether the server user has `loginctl enable-linger` set, which is what lets a `systemd --user` timer keep running without an active login session. Given the PR states the timer is already "active, 60 s cadence" on the live server, this is presumably already handled outside the repo — worth a one-line mention in `deploy/README.md`'s troubleshooting section for the next person who tries to reproduce this setup on a different host, but not a blocker. No correctness or security issues found in the shell scripts, workflow YAML, or compose changes. --- ⏱ 2m 46s · 17 turns · tokens: 90.3k in / 13.0k out (+1716.2k cache-read) 📜 [Full session transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/Ponente/pr-72/run-175.md) <!-- claude-verdict:approve --> <!-- claude-reviewed-head:5894594d30b22202342197503db4357a76199ba3 -->

Grandiras merged commit 7b1e998c7a into main 2026-08-08 01:53:51 +00:00

Grandiras referenced this pull request from a commit 2026-08-08 01:53:52 +00:00

Merge pull request 'Run a dev stack beside production, and write both flows down' (#72) from issue-69-dev-environment into main

Grandiras deleted branch issue-69-dev-environment 2026-08-08 01:54:31 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

claude-reviewer

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Grandiras/Ponente!72

No description provided.