Skip to content
View as .md

Durable streams

Append-only JSON logs addressed by URL, served in production by our own apps/durable-streams on Fly. Not Cloudflare Durable Objects — they share a word and nothing else (a durable stream is byte storage; a Durable Object is an isolate with state). This page is protocol substrate; arbe’s entry, payload, and client contracts live in streams.

PUT  /stream/{path}                     # create (JSON mode by default)
POST /stream/{path}                     # append (returns Stream-Next-Offset header)
GET  /stream/{path}?offset=X            # read range (JSON array)
GET  /stream/{path}?offset=X&live=long-poll | live=sse   # tail

Sentinels: -1 (full replay), now (tail). Historical reads are immutable byte ranges, CDN-cacheable. Live SSE cycles ~60s for CDN connection collapsing — clients reconnect from the last control event’s streamNextOffset.

Offsets are opaque. Store them; never parse, construct, compare or sort them. Five consequences, each one a bug someone will otherwise write:

  • No ordering guarantee. Today’s token looks like <read-seq>_<byte-offset>, which tempts a lexicographic or numeric compare. Neither is safe: the vendor documents no ordering, and says the format “may change between server versions”.
  • One offset per read batch, not per entry. A read of 40 entries returns 40 items and 1 offset. You cannot label an entry with its own native position — which is why the thread director numbers entries itself (packages/core/schemas/director.ts).
  • Only echo offsets back. A fabricated one can 400 — offset=0 looks like “from the start” and is not: Electric Cloud tolerated it, the upstream server answers 400 invalid offset (which the www proxy relays as a 502 the tail loop retries forever). Pass back what a HEAD, GET or POST returned, or the -1 / now sentinels, nothing else.
  • A future offset may be silently clamped by the server. A strict tailStream caller detects this from its first long-poll response: when an empty, caught-up response replaces the requested offset, the client rejects it. Generic follow/resume consumers remain forgiving, and no code interprets or orders either token.
  • Future-offset behavior is backend-specific and accepted as best-effort (decided 2026-08-30, after probing both backends). The vendored dev server clamps to head (empty response, real head as Stream-Next-Offset); the production binary replays from the stream start and echoes the requested seq back in Stream-Next-Offset. Protocol §5.6 actually says a catch-up read past the tail SHOULD echo the requested offset — both backends violate it, and forks in the same protocol MUST 400 on a past-tail offset, so upstream may tighten this someday. We do not chase it: --from with a future offset means “somewhere at or before the head, best-effort”, the shipped rejectUnhonoredFromOffset guard on thread entries read stays as the UX guard, and no positional validation is built client-side. (Upstream concepts.md does document same-stream offsets as lexicographically sortable, if a real check is ever needed — we still treat tokens as opaque.)
  • HEAD gives the tail offset without reading the stream.

Source: @durable-streams/client — “Offsets are opaque tokens - clients MUST NOT interpret the format” (dist/index.d.ts), format stability (skills/getting-started/SKILL.md), fabricated values (skills/forking/SKILL.md).

arbe-thread-{threadId}     # one per thread (threadStreamId() in @arbe/core/schemas/thread.ts)
arbe-house-{houseId}       # one per house — collection invalidation events (houseStreamId() in @arbe/core/schemas/house.ts)

Each item is an ArbeThreadEntry; its entry and payload contract live in streams. A house stream’s items are HouseStreamEvents ({collection, id?}): a www write route appends one after its DB write, and browsers tail the house stream to refetch the matching query collection — see data/sync. @arbe/streams/client provides the two transport clients (raw service + scoped proxy); @arbe/core/entries is the thread-aware consumer (ensureThreadStream, postThreadEntry, readThreadEntries) used from CLI, www, and dispatch. The web chat UI reads and writes via @arbe/core/client, decoded by apps/www/src/lib/chat-stream.ts and rendered by Chat.svelte.

Worth knowing:

  • Idempotent producers — Producer-Id / Producer-Epoch / Producer-Seq headers. Epoch fences zombies with 403; a retry of an accepted (id, epoch, seq) returns a dedup’d success. IdempotentProducer in the TS client handles batching and pipelining. Producer-epoch fencing is separate from the Postgres lease used by the thread director.
  • Forking — PUT with Stream-Forked-From: <path> branches a new independent stream at Stream-Fork-Offset (defaults to source tail) without copying history. Deleting the source soft-deletes it until the last fork is gone. Fits conversation branching, deterministic replay, and producer handoff.
  • Closure — Stream-Closed: true on a final POST seals a stream. Once closed it stays closed; reads still work.
  • TTL — Stream-TTL (relative seconds) or Stream-Expires-At (RFC 3339), mutually exclusive at create.
  • Retention — servers may drop old data. On 410 Gone, reset to -1 or now.
  • Live modes — ?live=sse for JSON/text streams, ?live=long-poll for binary or simple request-response.

Delete is hard. deleteThread (called by arbe thread delete and DELETE /api/threads/:id) drops the durable stream first, then the row — so a stream-side failure leaves the row in place for retry rather than orphaning a stream. deleteHouse snapshots child thread ids, lets the row delete cascade (FK on house), then sweeps each thread’s stream plus the house’s own stream best-effort — a stream-side hiccup is logged, never strands the row. A house whose stream is missing is not a special case for a reader: GET /api/houses/:id/stream calls ensureHouseStream (packages/core/house-events.ts) before proxying, so an untouched house answers an empty, caught-up stream. House streams are created with a seven-day Stream-TTL, which the server measures from the last read or write; the append path memoizes the create per process and forgets the memo when an append fails, so an expired stream is recreated on the next write.

arbe uses only the protocol, @durable-streams/client, and the TanStack AI transport for chat; the wider ecosystem lives upstream. Production runs apps/durable-streams, the upstream durable-streams-server binary (Caddy plugin, LMDB on a Fly volume) at https://arbe-durable-streams.fly.dev/v1/stream. Switching backends is a PUBLIC_DURABLE_STREAMS_URL change in www and backstage together — streams are not copied between backends, and a split pair breaks replies. The dev-grade backend is packages/streams/gateway.ts, a single-token proxy (DS_SECRET) over the embedded server.ts, which also backs the package’s tests and demo/ scripts. Electric Cloud hosted the streams until 2026-08-25 and is closing down.

Code: @arbe/streams/client, packages/streams/{gateway,server}.ts, @arbe/core/entries, packages/core/schemas/stream-entries/thread.ts, packages/core/schemas/thread.ts (threadStreamId), packages/core/schemas/stream-entries/house.ts, packages/core/house-events.ts (appendHouseEvent). Upstream docs: durable-streams.com, protocol spec.
See streams, threads, system/dispatch.