feat(serialization)!: forward-compatible models — open unions and unknown-field round-tripping #115

New issue

Closed

opened 2026-09-24 12:28:56 +00:00 by Grandiras · 0 comments

Grandiras commented

2026-09-24 12:28:56 +00:00

Owner

Part of the v0.7.0 update cycle. Lexicon unions are open by spec, but the SDK's are closed, and it has no way to round-trip unknown fields. One unrecognized $type anywhere in a response fails the whole call, and read-modify-write drops data.

Live bugs this fixes

Gallery posts crash feeds. app.bsky.embed.gallery (upstream since 2026-06-03) is not registered on EmbedBase / EmbedView. One gallery post fails the whole timeline, feed, author-feed, thread, search or getPosts page with JsonException. In a 40-second Jetstream sample, 0.27% of new posts used it, which is about 1 in 8 fifty-post pages.
Ozone crashes on unknown event types. Ozone emits 11 event types the SDK doesn't know (accountEvent, identityEvent, recordEvent, …), so QueryEventsAsync throws on a real Ozone.
Signed labels crash. Label.Sig is a bare byte[] without LexBytesJsonConverter, so every signed label ({"$bytes": …}) throws. QueryLabelsAsync against mod.bsky.app fails.
UpdateProfileAsync deletes profile data. It rebuilds ProfileRecord from 5 fields, dropping pinnedPost, pronouns, website, labels, joinedViaStarterPack and anything newer. It also writes without swapRecord.

Scope

Open unions: every [JsonPolymorphic] union base gets an Unknown{Base} variant carrying $type and the raw JSON, which serializes back byte-for-byte (for example an AtProtoUnion converter factory driven by the registered variants). Unions the Lexicon marks closed: true stay strict. Unions currently exposed as raw JsonElement (FeedViewPost.Reason, PostView.Threadgate, …) are not retyped here; the lexicon-fidelity issue does that. Leave Lexicon/Com/AtProto/Space/* and SimpleSpace/* unions for the Spaces work (a parallel PR is changing those files), and note it in the PR.
Unknown fields: a base (for example LexObject with [JsonExtensionData] IDictionary<string, JsonElement>? ExtensionData) for records and views, including AtProtoRecord, so unknown fields survive deserialize→serialize. Same exclusion for Space/SimpleSpace models.
LexiconTypeRegistry: registered union variants are currently ignored by every sub-client, because they only affect CreateOptions(), not AtProtoJsonDefaults.Options. Apply the registry to the options the clients actually use. Delete the write-only record-type registry, the unused _plugins list and the duplicate CreateOptions. Keep or remove LexiconPluginAttribute/LoadPluginsFromAssembly depending on whether they still do anything useful.
AtProtoJsonDefaults.Options: freeze it (MakeReadOnly()) after construction, and fix its XML remark that points to non-existent constructor parameters.
Register the gallery embed types (#main, #image, #view, #viewImage) on every union that references them (feed.post#embed, feed.defs#postView.embed, embed.record#viewRecord.embeds, embed.recordWithMedia#media / #view.media).
Label.Sig gets LexBytesJsonConverter. Share one pad-unpadded-base64 helper between LexBytesJsonConverter and DagCborEncoder, which also fixes the latent padding bug in JetstreamEventParser / JetstreamSegmentReader (Convert.FromBase64String rejects unpadded input).
UpdateProfileAsync: read-modify-write that preserves every field (via the extension data), writes with swapRecord, and retries a bounded number of times on InvalidSwap. Suggested shape: UpdateProfileAsync(Func<ProfileRecord, ProfileRecord> update, …). Add the missing ProfileRecord fields (pronouns, website, labels, joinedViaStarterPack).
Delete the no-op IAtProtoUnion marker if nothing needs it (keep the custom open-union test).

Acceptance

Tests: an unknown variant in every union deserializes to Unknown* and re-serializes identically; unknown record fields round-trip; a timeline page with a gallery post deserializes; a signed label deserializes; UpdateProfileAsync preserves unknown fields.
CHANGELOG: Fixed (the crashes and the data loss); Breaking changes (union base shapes, UpdateProfileAsync signature, frozen options) with migration notes.

Part of the v0.7.0 update cycle. Lexicon unions are **open** by spec, but the SDK's are closed, and it has no way to round-trip unknown fields. One unrecognized `$type` anywhere in a response fails the whole call, and read-modify-write drops data. ## Live bugs this fixes - **Gallery posts crash feeds.** `app.bsky.embed.gallery` (upstream since 2026-06-03) is not registered on `EmbedBase` / `EmbedView`. One gallery post fails the whole timeline, feed, author-feed, thread, search or `getPosts` page with `JsonException`. In a 40-second Jetstream sample, 0.27% of new posts used it, which is about 1 in 8 fifty-post pages. - **Ozone crashes on unknown event types.** Ozone emits 11 event types the SDK doesn't know (`accountEvent`, `identityEvent`, `recordEvent`, …), so `QueryEventsAsync` throws on a real Ozone. - **Signed labels crash.** `Label.Sig` is a bare `byte[]` without `LexBytesJsonConverter`, so every signed label (`{"$bytes": …}`) throws. `QueryLabelsAsync` against `mod.bsky.app` fails. - **`UpdateProfileAsync` deletes profile data.** It rebuilds `ProfileRecord` from 5 fields, dropping `pinnedPost`, `pronouns`, `website`, `labels`, `joinedViaStarterPack` and anything newer. It also writes without `swapRecord`. ## Scope - [ ] **Open unions:** every `[JsonPolymorphic]` union base gets an `Unknown{Base}` variant carrying `$type` and the raw JSON, which serializes back byte-for-byte (for example an `AtProtoUnion` converter factory driven by the registered variants). Unions the Lexicon marks `closed: true` stay strict. Unions currently exposed as raw `JsonElement` (`FeedViewPost.Reason`, `PostView.Threadgate`, …) are **not** retyped here; the lexicon-fidelity issue does that. **Leave `Lexicon/Com/AtProto/Space/*` and `SimpleSpace/*` unions for the Spaces work** (a parallel PR is changing those files), and note it in the PR. - [ ] **Unknown fields:** a base (for example `LexObject` with `[JsonExtensionData] IDictionary<string, JsonElement>? ExtensionData`) for records and views, including `AtProtoRecord`, so unknown fields survive deserialize→serialize. Same exclusion for Space/SimpleSpace models. - [ ] **`LexiconTypeRegistry`:** registered union variants are currently ignored by every sub-client, because they only affect `CreateOptions()`, not `AtProtoJsonDefaults.Options`. Apply the registry to the options the clients actually use. Delete the write-only record-type registry, the unused `_plugins` list and the duplicate `CreateOptions`. Keep or remove `LexiconPluginAttribute`/`LoadPluginsFromAssembly` depending on whether they still do anything useful. - [ ] **`AtProtoJsonDefaults.Options`:** freeze it (`MakeReadOnly()`) after construction, and fix its XML remark that points to non-existent constructor parameters. - [ ] Register the gallery embed types (`#main`, `#image`, `#view`, `#viewImage`) on every union that references them (`feed.post#embed`, `feed.defs#postView.embed`, `embed.record#viewRecord.embeds`, `embed.recordWithMedia#media` / `#view.media`). - [ ] `Label.Sig` gets `LexBytesJsonConverter`. Share one pad-unpadded-base64 helper between `LexBytesJsonConverter` and `DagCborEncoder`, which also fixes the latent padding bug in `JetstreamEventParser` / `JetstreamSegmentReader` (`Convert.FromBase64String` rejects unpadded input). - [ ] `UpdateProfileAsync`: read-modify-write that preserves every field (via the extension data), writes with `swapRecord`, and retries a bounded number of times on `InvalidSwap`. Suggested shape: `UpdateProfileAsync(Func<ProfileRecord, ProfileRecord> update, …)`. Add the missing `ProfileRecord` fields (`pronouns`, `website`, `labels`, `joinedViaStarterPack`). - [ ] Delete the no-op `IAtProtoUnion` marker if nothing needs it (keep the custom open-union test). ## Acceptance - Tests: an unknown variant in every union deserializes to `Unknown*` and re-serializes identically; unknown record fields round-trip; a timeline page with a gallery post deserializes; a signed label deserializes; `UpdateProfileAsync` preserves unknown fields. - CHANGELOG: `Fixed` (the crashes and the data loss); `Breaking changes` (union base shapes, `UpdateProfileAsync` signature, frozen options) with migration notes.

Grandiras referenced this issue

2026-09-24 12:39:03 +00:00

feat!: typed identifiers and uniform conventions across the public API #119

Grandiras referenced this issue

2026-09-24 12:39:05 +00:00

feat(records)!: record and custom-XRPC ergonomics, Bluesky helpers on client.Bsky #122

Grandiras referenced this issue

2026-09-24 12:39:09 +00:00

fix(spaces): protocol hardening, performance and a smaller server surface #133

Grandiras referenced this issue

2026-09-24 12:39:33 +00:00

Tracking: v0.7.0 update cycle — protocol catch-up, optimization, debloat, API reshaping #141