Ask whether a stack is stale by digest, not by uptime #77

Grandiras · 2026-08-08T02:30:13Z

Grandiras commented

2026-08-08 02:30:13 +00:00

Found by the maintainer within an hour of #75 merging: dev "didn't move" after the merge.

It was correct behaviour, and the docs were wrong about how to tell. #75 changed only docs and deploy scripts, so the build context was unchanged, build-and-push finished in 13 seconds off Docker's cache, :main kept the same digest, and the updater compared and correctly did nothing.

	registry	running
dev → `:main`	`2a5011bc…`	`2a5011bc…`	up to date
prod → `:latest`	`0d7f1382…`	`0d7f1382…`	up to date

Nothing was stale. But the release skill said "ponente-dev-app restarting minutes after the last merge is the signal. If it is hours old, either the Publish workflow failed or the timer did" — which sends you debugging a healthy system every time a merge is docs-only. That is a real defect in instructions written for agents who cannot see the stack.

The fix

Uptime is a fact about the build cache. The question anyone actually means is whether the running image matches what the tag points at.

scripts/check-stack-image.sh            # production, follows :latest
scripts/check-stack-image.sh --env dev  # the dev stack, follows :main

It compares the registry's digest for the tag against the container's, and needs no credentials — the registry serves the manifest to an anonymous token, which is the same reason the server can pull without a login.

It also reports a pin separately rather than as staleness. Production sitting on a version tag after a rollback is the other situation where somebody concludes the timer is broken, and it deserves to say so in its own words instead of failing.

Verified

prod: 0d7f1382 == 0d7f1382, "Up to date"
dev: 2a5011bc == 2a5011bc, "Up to date"
the pin branch genuinely reads IMAGE_TAG off the server (ponente → latest, ponente-dev → main) rather than silently returning empty and skipping itself

Docs updated in three places that gave the uptime advice: Phase A step 2, Phase B step 4, the handover checklist, plus the Updating dev and troubleshooting sections of deploy/README.md. The 13-second build-and-push is called out as the tell, since that is the artefact you actually see.

🤖 Generated with Claude Code

Found by the maintainer within an hour of #75 merging: dev "didn't move" after the merge. **It was correct behaviour, and the docs were wrong about how to tell.** #75 changed only docs and deploy scripts, so the build context was unchanged, `build-and-push` finished in **13 seconds** off Docker's cache, `:main` kept the same digest, and the updater compared and correctly did nothing. | | registry | running | | |---|---|---|---| | dev → `:main` | `2a5011bc…` | `2a5011bc…` | up to date | | prod → `:latest` | `0d7f1382…` | `0d7f1382…` | up to date | Nothing was stale. But the release skill said *"`ponente-dev-app` restarting minutes after the last merge is the signal. If it is hours old, either the Publish workflow failed or the timer did"* — which sends you debugging a healthy system every time a merge is docs-only. That is a real defect in instructions written for agents who cannot see the stack. ## The fix Uptime is a fact about the build cache. The question anyone actually means is whether the running image matches what the tag points at. ```bash scripts/check-stack-image.sh # production, follows :latest scripts/check-stack-image.sh --env dev # the dev stack, follows :main ``` It compares the registry's digest for the tag against the container's, and needs no credentials — the registry serves the manifest to an anonymous token, which is the same reason the server can pull without a login. It also reports a **pin** separately rather than as staleness. Production sitting on a version tag after a rollback is the *other* situation where somebody concludes the timer is broken, and it deserves to say so in its own words instead of failing. ## Verified - prod: `0d7f1382` == `0d7f1382`, "Up to date" - dev: `2a5011bc` == `2a5011bc`, "Up to date" - the pin branch genuinely reads `IMAGE_TAG` off the server (`ponente` → `latest`, `ponente-dev` → `main`) rather than silently returning empty and skipping itself Docs updated in three places that gave the uptime advice: Phase A step 2, Phase B step 4, the handover checklist, plus the *Updating dev* and troubleshooting sections of `deploy/README.md`. The 13-second `build-and-push` is called out as the tell, since that is the artefact you actually see. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Grandiras added 1 commit

2026-08-08 02:30:13 +00:00

Ask whether a stack is stale by digest, not by uptime

CI / build (pull_request) Successful in 36s

Details

53ef1be3b8

The docs told you to read `ponente-dev-app`'s uptime and to treat an old
container as a failed timer. That is wrong, and it produced its own false
alarm within the hour: PR #75 changed only docs and deploy scripts, so the
build context was unchanged, `build-and-push` finished in 13 seconds off
Docker's cache, `:main` kept the same digest, and the updater correctly did
nothing. Both stacks were exactly on the tags they follow and it read as a
stall.

Uptime is a fact about the build cache. The question anyone actually means is
whether the running image matches what the tag points at, so ask that:

  scripts/check-stack-image.sh            # production, follows :latest
  scripts/check-stack-image.sh --env dev  # the dev stack, follows :main

It compares the registry's digest against the container's — no credentials
needed, the registry serves the manifest to an anonymous token — and reports a
**pin** separately, because production sitting on a version tag after a
rollback is precisely the other situation where somebody concludes the timer is
broken.

Verified against both live stacks: prod 0d7f1382 == 0d7f1382, dev 2a5011bc ==
2a5011bc, and the pin branch reads IMAGE_TAG off the server rather than
silently returning empty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

claude-bot commented

2026-08-08 02:30:16 +00:00

🤖 Reviewing this PR — follow along in the action run; the reply will link the full session transcript.

📜 Live transcript — updating every ~5 minutes while the task runs.

🤖 Reviewing this PR — follow along in [the action run](https://git.grandiras.net/Grandiras/claude-bot/actions/runs/177); the reply will link the full session transcript. 📜 [Live transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/Ponente/pr-77/run-177.md) — updating every ~5 minutes while the task runs.

claude-reviewer reviewed

2026-08-08 02:32:55 +00:00

claude-reviewer left a comment

💬 Verdict: comment — notes only, nothing blocking.

The core fix (compare registry digest vs running image digest, instead of container uptime) is sound and well-motivated, and the docs updates consistently point at the new script everywhere the old uptime advice appeared. One robustness gap in the new pin-detection code path is worth fixing before/soon after merge, but it doesn't undermine the primary digest-comparison logic that the PR verified works.

⏱ 2m 25s · 11 turns · tokens: 66.6k in / 13.1k out (+828.0k cache-read)
📜 Full session transcript

💬 **Verdict: comment** — notes only, nothing blocking. The core fix (compare registry digest vs running image digest, instead of container uptime) is sound and well-motivated, and the docs updates consistently point at the new script everywhere the old uptime advice appeared. One robustness gap in the new pin-detection code path is worth fixing before/soon after merge, but it doesn't undermine the primary digest-comparison logic that the PR verified works. --- ⏱ 2m 25s · 11 turns · tokens: 66.6k in / 13.1k out (+828.0k cache-read) 📜 [Full session transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/Ponente/pr-77/run-177.md)

.claude/skills/release/scripts/check-stack-image.sh

					
				@ -0,0 +67,4 @@

				# failing: production sitting on a version tag after a rollback is exactly the

				# situation where somebody is about to conclude the timer is broken.

				env_tag="$(ssh -o BatchMode=yes "podman@$( \

				    podman system connection ls --format json 2>/dev/null \

The pin check builds its own SSH target instead of reusing the connection-resolution logic in prod-podman.sh (and deploy/deploy.sh), and drops two things those scripts are careful about:

Hardcodes the SSH user as podman@ rather than reading it from the podman connection URI (as prod-podman.sh/deploy.sh do — deliberately leaving the user off when the URI doesn't specify one, so ~/.ssh/config can decide).
Omits -p $SSH_PORT entirely — prod-podman.sh explicitly extracts u.port or 22 from the connection and passes -p "$SSH_PORT" to every ssh invocation, which only makes sense if a non-default port is a real possibility here.

Because the whole ssh ... || true pipeline is swallowed (2>/dev/null || true), a mismatch in either of these silently yields an empty env_tag, which the script then treats identically to "not pinned." That's the exact failure mode the PR's own "Verified" section says was checked ("rather than silently returning empty and skipping itself") — but it was only verified against whatever connection config happens to be in use in this environment, not against a differently-configured one. A production stack that's genuinely pinned after a rollback would then be reported as STALE instead of PINNED, which is precisely the false alarm this PR sets out to eliminate on the uptime side.

Worth reusing the same connection-parsing logic prod-podman.sh already has (or factoring it out) rather than a second, less careful reimplementation.

The pin check builds its own SSH target instead of reusing the connection-resolution logic in `prod-podman.sh` (and `deploy/deploy.sh`), and drops two things those scripts are careful about: - **Hardcodes the SSH user as `podman@`** rather than reading it from the podman connection URI (as `prod-podman.sh`/`deploy.sh` do — deliberately leaving the user off when the URI doesn't specify one, so `~/.ssh/config` can decide). - **Omits `-p $SSH_PORT`** entirely — `prod-podman.sh` explicitly extracts `u.port or 22` from the connection and passes `-p "$SSH_PORT"` to every ssh invocation, which only makes sense if a non-default port is a real possibility here. Because the whole `ssh ... || true` pipeline is swallowed (`2>/dev/null || true`), a mismatch in either of these silently yields an empty `env_tag`, which the script then treats identically to "not pinned." That's the exact failure mode the PR's own "Verified" section says was checked ("rather than silently returning empty and skipping itself") — but it was only verified against whatever connection config happens to be in use in this environment, not against a differently-configured one. A production stack that's genuinely pinned after a rollback would then be reported as `STALE` instead of `PINNED`, which is precisely the false alarm this PR sets out to eliminate on the uptime side. Worth reusing the same connection-parsing logic `prod-podman.sh` already has (or factoring it out) rather than a second, less careful reimplementation.

Grandiras referenced this pull request from a commit

2026-08-08 02:36:47 +00:00

Share one connection resolver, and never let it fail silently

Grandiras added 1 commit

2026-08-08 02:36:47 +00:00

Share one connection resolver, and never let it fail silently

CI / build (pull_request) Successful in 36s

Details

6186a83cfd

Review finding on #77, and a correct one: check-stack-image.sh reimplemented
the podman-connection parsing instead of reusing prod-podman.sh's, and the copy
was worse in two ways — it hardcoded the SSH user as `podman@` rather than
letting a URI without one fall through to ~/.ssh/config, and it dropped
`-p $SSH_PORT` entirely.

The reason that mattered is the third thing: the call was wrapped in
`2>/dev/null || true`, so either mismatch came back as an empty answer that the
script read as "not pinned". A production stack deliberately held on a version
after a rollback would then have been reported STALE — the exact false alarm
this PR exists to remove, reintroduced on the other side.

So the parsing moves to lib/podman-connection.sh and both scripts source it.
deploy/deploy.sh keeps its own copy on purpose and says so: it is the primary
operational tool and has to work from a plain checkout with no skill directory
beside it.

The silent-degradation half is fixed separately, because sharing the resolver
would not have fixed it: an unreachable host now prints a WARNING and says the
pin state is unknown, rather than defaulting to "armed". Reaching the host but
finding no IMAGE_TAG is reported as its own case.

Also stopped shipping the whole env file over the wire to answer a question
about one key — the reachability probe used to `cat` it and discard the output.

Verified: pin detection fires (dev temporarily pinned → "PINNED to v2.3.0",
and the updater independently agreed "leaving it alone"), restore is clean,
a bad PONENTE_CONNECTION exits 1 loudly from both scripts rather than
answering wrongly, and prod-podman.sh still works after the refactor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Grandiras commented

2026-08-08 02:37:16 +00:00

Fixed in 6186a83 — the finding was right on all three counts, and the third was the important one.

The parsing now lives in scripts/lib/podman-connection.sh and both prod-podman.sh and check-stack-image.sh source it, so the hardcoded podman@ and the missing -p $SSH_PORT are gone rather than fixed twice. deploy/deploy.sh deliberately keeps its own copy — it is the primary operational tool and has to work from a plain checkout with no skill directory beside it — and the library now says so, so the duplication reads as a decision instead of an oversight.

Sharing the resolver would not have fixed the silent degradation, so that is handled separately. An unreachable host now prints a WARNING and reports the pin state as unknown, instead of falling through to "not pinned". Reaching the host but finding no IMAGE_TAG is its own case. That was the part that would have turned a genuinely-pinned production stack into a STALE report.

Two things beyond the finding:

The reachability probe used to cat the entire env file and discard it, putting every secret in it on the wire to answer a question about one key. It now greps the single line.
Tested the paths rather than reasoning about them: dev temporarily pinned → PINNED to v2.3.0, with the updater independently agreeing (ponente-dev is pinned to v2.3.0; leaving it alone), then restored clean; a bad PONENTE_CONNECTION exits 1 loudly from both scripts instead of answering wrongly; prod-podman.sh still drives the host after the refactor.

You were right that the original "Verified" claim only held for the connection config in front of me.

Fixed in 6186a83 — the finding was right on all three counts, and the third was the important one. The parsing now lives in `scripts/lib/podman-connection.sh` and both `prod-podman.sh` and `check-stack-image.sh` source it, so the hardcoded `podman@` and the missing `-p $SSH_PORT` are gone rather than fixed twice. `deploy/deploy.sh` deliberately keeps its own copy — it is the primary operational tool and has to work from a plain checkout with no skill directory beside it — and the library now says so, so the duplication reads as a decision instead of an oversight. **Sharing the resolver would not have fixed the silent degradation, so that is handled separately.** An unreachable host now prints a WARNING and reports the pin state as *unknown*, instead of falling through to "not pinned". Reaching the host but finding no `IMAGE_TAG` is its own case. That was the part that would have turned a genuinely-pinned production stack into a STALE report. Two things beyond the finding: - The reachability probe used to `cat` the entire env file and discard it, putting every secret in it on the wire to answer a question about one key. It now greps the single line. - Tested the paths rather than reasoning about them: dev temporarily pinned → `PINNED to v2.3.0`, with the updater independently agreeing (`ponente-dev is pinned to v2.3.0; leaving it alone`), then restored clean; a bad `PONENTE_CONNECTION` exits 1 loudly from *both* scripts instead of answering wrongly; `prod-podman.sh` still drives the host after the refactor. You were right that the original "Verified" claim only held for the connection config in front of me.

Grandiras commented

2026-08-08 02:48:37 +00:00

@claude-bot /review

claude-bot commented

2026-08-08 02:48:40 +00:00

🤖 Reviewing this PR — follow along in the action run; the reply will link the full session transcript.

📜 Live transcript — updating every ~5 minutes while the task runs.

🤖 Reviewing this PR — follow along in [the action run](https://git.grandiras.net/Grandiras/claude-bot/actions/runs/178); the reply will link the full session transcript. 📜 [Live transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/Ponente/pr-77/run-178.md) — updating every ~5 minutes while the task runs.

claude-reviewer approved these changes

2026-08-08 02:49:30 +00:00

claude-reviewer left a comment

✅ Verdict: approve — no blocking issues found.

The pin-detection robustness gap I flagged in my previous pass is fixed: the SSH invocation for reading IMAGE_TAG off the server now goes through the shared resolve_connection() in the new lib/podman-connection.sh (sourced by both check-stack-image.sh and the refactored prod-podman.sh), using $SSH_TARGET/$SSH_PORT derived from the actual podman connection URI instead of a hardcoded podman@ user and a missing port. That was the exact failure mode that could have silently misreported a pinned production stack as STALE. The extraction into a shared lib is a clean fix rather than a patch — it also removes the duplicated connection-parsing logic that used to live only in prod-podman.sh, and the comments in the new lib file explain why the duplication was a bug rather than tolerable drift. Docs updates are consistent throughout (skill, README, bundled-scripts list) and match the new script's behavior. No remaining issues.

⏱ 0m 35s · 6 turns · tokens: 49.6k in / 2.6k out (+417.9k cache-read)
📜 Full session transcript

✅ **Verdict: approve** — no blocking issues found. The pin-detection robustness gap I flagged in my previous pass is fixed: the SSH invocation for reading `IMAGE_TAG` off the server now goes through the shared `resolve_connection()` in the new `lib/podman-connection.sh` (sourced by both `check-stack-image.sh` and the refactored `prod-podman.sh`), using `$SSH_TARGET`/`$SSH_PORT` derived from the actual podman connection URI instead of a hardcoded `podman@` user and a missing port. That was the exact failure mode that could have silently misreported a pinned production stack as `STALE`. The extraction into a shared lib is a clean fix rather than a patch — it also removes the duplicated connection-parsing logic that used to live only in `prod-podman.sh`, and the comments in the new lib file explain why the duplication was a bug rather than tolerable drift. Docs updates are consistent throughout (skill, README, bundled-scripts list) and match the new script's behavior. No remaining issues. --- ⏱ 0m 35s · 6 turns · tokens: 49.6k in / 2.6k out (+417.9k cache-read) 📜 [Full session transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/Ponente/pr-77/run-178.md)

Grandiras merged commit a28017c1c0 into main

2026-08-08 02:49:36 +00:00

Grandiras referenced this pull request from a commit

2026-08-08 02:49:38 +00:00

Merge pull request 'Ask whether a stack is stale by digest, not by uptime' (#77) from fix-staleness-check-by-digest into main

Grandiras deleted branch fix-staleness-check-by-digest

2026-08-08 02:49:43 +00:00

claude-bot referenced this pull request

2026-08-18 13:53:32 +00:00

Let's push out a new release to prod #106

Sign in to join this conversation.

No reviewers