Add support for Jetstream V2

Grandiras commented

2026-08-13 20:40:50 +00:00

Owner

No description provided.

claude-bot was assigned by Grandiras

2026-08-14 01:25:05 +00:00

claude-bot commented

2026-08-14 01:25:09 +00:00

Collaborator

🤖 On it — follow along in the action run; the reply will link the full session transcript.

📜 Live transcript — updating every ~5 minutes while the task runs.

🤖 On it — follow along in [the action run](https://git.grandiras.net/Grandiras/claude-bot/actions/runs/179); the reply will link the full session transcript. 📜 [Live transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/ATProto.NET/issue-83/run-179.md) — updating every ~5 minutes while the task runs.

claude-bot referenced this issue from a commit

2026-08-14 01:42:06 +00:00

feat: support the Jetstream v2 wire protocol

claude-bot referenced this issue from a pull request that will close it,

2026-08-14 01:42:41 +00:00

Claude: changes for issue #83 #84

claude-bot commented

2026-08-14 01:42:41 +00:00

Collaborator

Pushed. Here's the reply:

Added support for the Jetstream v2 wire protocol on branch claude/issue-83.

What v2 is

Jetstream's second wire serves at /xrpc/network.bsky.jetstream.subscribeEvents under the xrpc.v1.json subprotocol (atproto proposal 0015), and differs from the frozen v1 wire in almost every particular: a self-describing {"$type":"message","payload":{…}} envelope per frame, flat commit fields instead of a nested commit object, filters renamed collections/dids and joined by a new kinds, a monotonic sequence number as the cursor instead of a time_us timestamp, a sync event kind v1 never emits, and out-of-band #info/error frames.

JetstreamClient and JetstreamConsumer now speak both, selected by the new JetstreamConsumerOptions.Protocol.

What changed

Protocol defaults to JetstreamProtocol.V1 — every existing configuration keeps its endpoint, parameter names, and timestamp cursor with no change. The legacy jetstream1/jetstream2 hosts serve v1 only; the two v2 hosts serve both.
JetstreamEndpoints — UsEast/UsWest (v2) and the four legacy hosts, instead of pasting host names.
WantedKinds — the v2 kinds filter. A collection filter constrains commit events only (identity/account/sync flow regardless, on both protocols), so combining it with a WantedKinds list that excludes Commit is a filter that could never apply — that now throws before the socket opens rather than coming back as an HTTP 400.
JetstreamSyncEvent — resync marker with Rev and the commit CAR in Blocks, decoded from the wire's $bytes wrapper.
JetstreamEvent.Cursor — the sequence number. Also populated on the v1 wire from the cursor field the v2 hosts added to it (null against a legacy instance). Plus Timestamp as a DateTimeOffset; on v2 both derive from the RFC 3339 time, whose six fractional digits convert exactly.
Cursor handling follows the protocol — the consumer tracks LastCursor alongside LastTimeUs, persists whichever the protocol uses, and reconnects at exactly the last sequence number, since a v2 cursor is replayed inclusively and ReconnectRewind has nothing to rewind past.
A rejected subscription is no longer retried in a loop — v2 validates pre-upgrade and answers a stale cursor (CursorTooOld), retired dictionary, or malformed filter with an HTTP 400. That surfaces as the new JetstreamConnectException (StatusCode, IsRetryable); the consumer persists progress and rethrows, because retrying is futile and dropping the cursor would silently skip the gap you're meant to backfill.
OnInfo / OnStreamError — v2's #info advisories and terminal error frames are neither events nor exceptions; they're logged and passed to these.
JetstreamDictionaryClient — v2 compression is dictionary-versioned, so it fetches getZstdDictionary and reads the ID out of the zstd dictionary's own header, yielding both halves of the new ZstdDictionaryId + Decompressor pair in one call.
JetstreamEventParser.ParseFrame(json, protocol) is the new entry point returning a JetstreamFrame; the existing Parse(...) overloads are unchanged.

Deliberately out of scope

v2 hosts also serve an authenticated archive/replay API (planSnapshot/listSegments/getSegment/getBlock over a columnar .jss segment format, with cutover to the live tail). That's a much larger surface than the live tail — it needs a .jss decoder and API-key metering — so I implemented the live tail only. docs/jetstream.md says so explicitly and points at Bluesky's own SDKs for the archive portion.

Verification

dotnet build across the solution: 0 warnings, 0 errors.
dotnet test tests/ATProtoNet.Tests/: 1237 passed, 0 failed, 5 gated skips (same as baseline) — up from 1228, with 4 new test files covering v2 parsing, URL building, consumer cursor semantics, and the dictionary client.
Live against the production instances (jetstream.us-east / jetstream.us-west): commit/identity/account parsing with correct seq, microsecond time, URI, rev and CID; the kinds filter (zero commits over 25 s where the unfiltered stream runs hundreds/sec); inclusive sequence-number resume (asked 24707154431, got exactly that); the pre-upgrade 400 for cursor=1 mapping to a non-retryable JetstreamConnectException; the dictionary fetch (id 20260811, 64 KiB); and confirmation that the v1 wire on a v2 host does carry the cursor field the SDK now surfaces. No sync event happened to occur during the window — that path is covered by unit tests only.

CHANGELOG.md updated under [Unreleased] → Added; docs updated in docs/jetstream.md (new Protocol Versions, Advisory and error frames, and Historical Replay sections), docs/api-reference.md, docs/index.md, and README.md.

⏱ 17m 9s · 66 turns · tokens: 179.7k in / 81.3k out (+7240.6k cache-read)
📜 Full session transcript
→ Opened PR: #84

Pushed. Here's the reply: --- Added support for the **Jetstream v2 wire protocol** on branch **`claude/issue-83`**. ## What v2 is Jetstream's second wire serves at `/xrpc/network.bsky.jetstream.subscribeEvents` under the `xrpc.v1.json` subprotocol (atproto proposal 0015), and differs from the frozen v1 wire in almost every particular: a self-describing `{"$type":"message","payload":{…}}` envelope per frame, flat commit fields instead of a nested `commit` object, filters renamed `collections`/`dids` and joined by a new `kinds`, a monotonic **sequence number** as the cursor instead of a `time_us` timestamp, a `sync` event kind v1 never emits, and out-of-band `#info`/`error` frames. `JetstreamClient` and `JetstreamConsumer` now speak both, selected by the new `JetstreamConsumerOptions.Protocol`. ## What changed - **`Protocol` defaults to `JetstreamProtocol.V1`** — every existing configuration keeps its endpoint, parameter names, and timestamp cursor with no change. The legacy `jetstream1`/`jetstream2` hosts serve v1 only; the two v2 hosts serve both. - **`JetstreamEndpoints`** — `UsEast`/`UsWest` (v2) and the four legacy hosts, instead of pasting host names. - **`WantedKinds`** — the v2 `kinds` filter. A collection filter constrains *commit* events only (identity/account/sync flow regardless, on both protocols), so combining it with a `WantedKinds` list that excludes `Commit` is a filter that could never apply — that now throws before the socket opens rather than coming back as an HTTP 400. - **`JetstreamSyncEvent`** — resync marker with `Rev` and the commit CAR in `Blocks`, decoded from the wire's `$bytes` wrapper. - **`JetstreamEvent.Cursor`** — the sequence number. Also populated on the v1 wire from the `cursor` field the v2 hosts added to it (null against a legacy instance). Plus `Timestamp` as a `DateTimeOffset`; on v2 both derive from the RFC 3339 `time`, whose six fractional digits convert exactly. - **Cursor handling follows the protocol** — the consumer tracks `LastCursor` alongside `LastTimeUs`, persists whichever the protocol uses, and reconnects at exactly the last sequence number, since a v2 cursor is replayed *inclusively* and `ReconnectRewind` has nothing to rewind past. - **A rejected subscription is no longer retried in a loop** — v2 validates pre-upgrade and answers a stale cursor (`CursorTooOld`), retired dictionary, or malformed filter with an HTTP 400. That surfaces as the new `JetstreamConnectException` (`StatusCode`, `IsRetryable`); the consumer persists progress and rethrows, because retrying is futile and dropping the cursor would silently skip the gap you're meant to backfill. - **`OnInfo` / `OnStreamError`** — v2's `#info` advisories and terminal `error` frames are neither events nor exceptions; they're logged and passed to these. - **`JetstreamDictionaryClient`** — v2 compression is dictionary-versioned, so it fetches `getZstdDictionary` and reads the ID out of the zstd dictionary's own header, yielding both halves of the new `ZstdDictionaryId` + `Decompressor` pair in one call. - `JetstreamEventParser.ParseFrame(json, protocol)` is the new entry point returning a `JetstreamFrame`; the existing `Parse(...)` overloads are unchanged. ## Deliberately out of scope v2 hosts also serve an **authenticated archive/replay API** (`planSnapshot`/`listSegments`/`getSegment`/`getBlock` over a columnar `.jss` segment format, with cutover to the live tail). That's a much larger surface than the live tail — it needs a `.jss` decoder and API-key metering — so I implemented the live tail only. `docs/jetstream.md` says so explicitly and points at Bluesky's own SDKs for the archive portion. ## Verification - `dotnet build` across the solution: **0 warnings, 0 errors**. - `dotnet test tests/ATProtoNet.Tests/`: **1237 passed, 0 failed**, 5 gated skips (same as baseline) — up from 1228, with 4 new test files covering v2 parsing, URL building, consumer cursor semantics, and the dictionary client. - **Live against the production instances** (`jetstream.us-east` / `jetstream.us-west`): commit/identity/account parsing with correct seq, microsecond time, URI, rev and CID; the `kinds` filter (zero commits over 25 s where the unfiltered stream runs hundreds/sec); inclusive sequence-number resume (asked `24707154431`, got exactly that); the pre-upgrade 400 for `cursor=1` mapping to a non-retryable `JetstreamConnectException`; the dictionary fetch (id `20260811`, 64 KiB); and confirmation that the v1 wire on a v2 host does carry the `cursor` field the SDK now surfaces. No `sync` event happened to occur during the window — that path is covered by unit tests only. `CHANGELOG.md` updated under `[Unreleased] → Added`; docs updated in `docs/jetstream.md` (new **Protocol Versions**, **Advisory and error frames**, and **Historical Replay** sections), `docs/api-reference.md`, `docs/index.md`, and `README.md`. --- ⏱ 17m 9s · 66 turns · tokens: 179.7k in / 81.3k out (+7240.6k cache-read) 📜 [Full session transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/ATProto.NET/issue-83/run-179.md) → Opened PR: https://git.grandiras.net/Grandiras/ATProto.NET/pulls/84

Grandiras commented

2026-08-14 01:46:02 +00:00

Author

Owner

@claude-bot Okay, but please open another issue for that archive/replay API, I wanna support it eventually.

claude-bot commented

2026-08-14 01:46:04 +00:00

Collaborator

🤖 On it — follow along in the action run; the reply will link the full session transcript.

📜 Live transcript — updating every ~5 minutes while the task runs.

🤖 On it — follow along in [the action run](https://git.grandiras.net/Grandiras/claude-bot/actions/runs/182); the reply will link the full session transcript. 📜 [Live transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/ATProto.NET/issue-83/run-182.md) — updating every ~5 minutes while the task runs.

claude-bot referenced this issue

2026-08-14 01:48:50 +00:00

Add support for the Jetstream v2 archive/replay API #85

claude-bot commented

2026-08-14 01:49:15 +00:00

Collaborator

Opened issue #85 — Add support for the Jetstream v2 archive/replay API (labelled enhancement), tracking the piece #84 left out.

Rather than just restating "the archive exists", I read Bluesky's Network Replay docs and the .jss data-layout spec and wrote the issue up as something implementable:

The three consumption modes (live / replay / snapshot) and which one #84 covered.
The replay flow in full — page planSnapshot pinning sealedTipSeq, re-plan while plannedThroughSeq < S, download per-segment mode: "segment" vs mode: "blocks", then connect the live socket once at ?cursor=S with seq dedup because the cursor is inclusive.
The .jss format — 256-byte header, length-prefixed zstd blocks, footer with block index and DID blooms; the columnar block layout down to the individual columns and the kind discriminator. This is the bulk of the work and the reason it didn't ride along.
Auth and metering — Bearer key, metering by wire bytes, 429 + Retry-After, and byte-offset Range resume, which needs to be designed in rather than bolted on.
Two semantics that bite if discovered late — sealed segments are immutable only between compactions (re-list and compare checksums), and the stream is folded rather than server-filtered, so writes must be idempotent on the at:// URI and account-level markers must not be dropped.
A sketch of the API surface fitting what #84 landed (JetstreamArchiveClient, JetstreamSegmentReader, JetstreamReplayConsumer, cursor reuse via IFirehoseCursorStore), a five-step phasing that puts the offline-testable .jss reader first, and four open design questions.

It also notes the failure mode where a long backfill lets S age out of the 36-hour lookback window — that already surfaces as the JetstreamConnectException from #84, so the handling is "re-enter the plan loop", not "retry".

No code changes and no branch this time; #84 is unchanged. Say the word if you'd like me to add a cross-reference to #85 from the Historical Replay section of docs/jetstream.md on claude/issue-83 before that PR merges.

⏱ 2m 54s · 23 turns · tokens: 41.5k in / 9.9k out (+732.7k cache-read)
📜 Full session transcript

Opened **[issue #85 — Add support for the Jetstream v2 archive/replay API](https://git.grandiras.net/Grandiras/ATProto.NET/issues/85)** (labelled `enhancement`), tracking the piece #84 left out. Rather than just restating "the archive exists", I read Bluesky's [Network Replay docs](https://bsky.network/docs/jetstream-replay) and the [`.jss` data-layout spec](https://github.com/bluesky-social/jetstream/tree/main/docs#3-data-layout) and wrote the issue up as something implementable: - **The three consumption modes** (live / replay / snapshot) and which one #84 covered. - **The replay flow in full** — page `planSnapshot` pinning `sealedTipSeq`, re-plan while `plannedThroughSeq < S`, download per-segment `mode: "segment"` vs `mode: "blocks"`, then connect the live socket once at `?cursor=S` with seq dedup because the cursor is inclusive. - **The `.jss` format** — 256-byte header, length-prefixed zstd blocks, footer with block index and DID blooms; the columnar block layout down to the individual columns and the `kind` discriminator. This is the bulk of the work and the reason it didn't ride along. - **Auth and metering** — Bearer key, metering by wire bytes, `429` + `Retry-After`, and byte-offset `Range` resume, which needs to be designed in rather than bolted on. - **Two semantics that bite if discovered late** — sealed segments are immutable *only between compactions* (re-list and compare checksums), and the stream is folded rather than server-filtered, so writes must be idempotent on the `at://` URI and account-level markers must not be dropped. - A sketch of the API surface fitting what #84 landed (`JetstreamArchiveClient`, `JetstreamSegmentReader`, `JetstreamReplayConsumer`, cursor reuse via `IFirehoseCursorStore`), a five-step phasing that puts the offline-testable `.jss` reader first, and four open design questions. It also notes the failure mode where a long backfill lets `S` age out of the 36-hour lookback window — that already surfaces as the `JetstreamConnectException` from #84, so the handling is "re-enter the plan loop", not "retry". No code changes and no branch this time; #84 is unchanged. Say the word if you'd like me to add a cross-reference to #85 from the *Historical Replay* section of `docs/jetstream.md` on `claude/issue-83` before that PR merges. --- ⏱ 2m 54s · 23 turns · tokens: 41.5k in / 9.9k out (+732.7k cache-read) 📜 [Full session transcript](https://git.grandiras.net/Grandiras/claude-bot/src/branch/claude-logs/transcripts/Grandiras/ATProto.NET/issue-83/run-182.md)

Grandiras closed this issue

2026-08-14 02:16:29 +00:00

Grandiras referenced this issue from a commit

2026-08-14 02:16:29 +00:00

Claude: changes for issue #83 (#84)

Grandiras referenced this issue

2026-08-21 14:04:14 +00:00

chore: release v0.6.0 #107