Skip to content
View as .md

One-daemon dispatch

Decision for arbe-9801 (2026-08-22). Status: fully shipped 2026-08-22 (arbe-228d, arbe-a16f, arbe-e1f9, arbe-448b). Turns and the speaker decision run in the conductor; celld and the inline path are deleted; local dev runs the same dispatcher on a per-dev queue; post_to_thread and wf-step ride the one enqueue. This document is now the historical record of why.

Reached by two routes independently: this session’s iteration from the arbe-9801 sketch, and a blind architecture review (fresh model, no access to .arbe/, given only the product goal and the observed symptoms) that converged on the same shape. Where they differed, the blind review’s version was simpler and won (per-dev queue over origin stamping; keep the existing fence over an advisory-lock rewrite; no interim smarter-fallback).

Goal

The product goal governs: group chat that feels fast and human — you post, the right bot replies quickly without being summoned by ritual, and silence is always explained. The design principle that follows, ranked above per-hop latency and above code reuse:

Variants of the dispatch pipeline are the primary cost. Every future special case (a new tool that posts cross-thread, a workflow step, a DM shortcut) must ride the one path, not add another. Debugging “why didn’t the bot reply” must never start with “which pipeline was this even on”.

Problem

“Which bot speaks and where does its turn run” has five answers today: the director-cell permit path (prod), the deterministic cell-down fallback (prod), the inline mention-only path (local dev), post_to_thread (inline even on prod — ambient bots never wake on cross-thread posts), and /api/wf/step’s drive machinery. Decider (celld), driver (conductor), and executor (www on Cloudflare Workers) are three processes on three availability domains, coordinated by two leases, a permit/claim protocol, and a fallback ladder — nearly all of which defends against network failures between them.

The observed pains all trace to the split, not to the semantics: ambient untestable locally (dev is a different pipeline), dispatch changes only show real behavior after deploy, long replies die mid-stream at the Cloudflare worker resource limit, and diagnosis starts with pipeline archaeology.

Premise correction to the original arbe-9801 text: a local post does NOT enqueue director_spawn for the prod conductor — it runs the inline path in the local worker (proven by inline- taskIds / MAX_INLINE_ATTEMPTS on its signals). The incident that motivated the task came through a prod-pointed surface. The lockout is the Worker-resident executor plus the shared queue consumer, not the shared DB.

Decision

Collapse to one pipeline: www appends and enqueues; the dispatcher decides and speaks; Postgres remembers.

any surface → POST /entries (www, Workers): append entry + director_spawn(queue) → NOTIFY
→ dispatcher daemon (Fly in prod, laptop in dev; bun + @arbe/core):
    claim task → execution_fence_claim (the ONE lease)
    → read stream tail + beat row → decide speaker (core/director pure fns;
      mention/forced path model-free, ambient = one cheap model call)
    → run the turn in-process (createThreadDispatcher, streams via bearer client)
    → commit beat cursor, release fence
Each bot reply enqueues the next trigger — no in-process recursion anywhere.

Survives: the durable stream as the only log; packages/core/dispatch/* (tool loop, turn, signals, reader copy); packages/core/director/* pure functions + tests — the locked beat/queue/narration semantics (2026-08-14) port for free; the Absurd dispatch queue, director_spawn, the pg_notify wake; the Postgres execution fence RPCs; submitThreadEntries (always spawn).

Changes: the workflow-conductor imports @arbe/core and executes decision + turn in its task handlers instead of driving www over HTTP. Verified feasible: everything a turn needs is env-derivable — actor-JWT minting is a secret + mintAgentJwt, streams is a bearer HTTP client, makeDispatchDeps is a plain function of those. Beat state (decidedThrough cursor, forced queue, sticky decision) becomes one small per-thread row mutated under the fence; thread-director.md itself notes the cell’s state is reconstructible from the stream, i.e. celld was never the source of truth.

Deleted: apps/thread-director (celld + cell, R2, restart ritual, proofs); the cell-client half of conductor director.ts (claim ladder, permits, activation clocks, fallback drive); the /api/director/* routes; /api/dispatch/run + /api/dispatch/narrate-failure and the held-open POST with its timeout pathology; WorkerResourceLimitError; DIRECTOR_SHARED_SECRET/DIRECTOR_CELL_URL/DIRECTOR_MASTER_KEY; the inline fan-out, kickDispatch, dispatchMode, and deterministicDirectorDecision as a selection path; the permit schema layer.

Time-to-first-token improves structurally: today’s prod path pays queue claim + cell round-trips + a www decision call + fence RPC + a second held-open POST before signal.dispatch.started. Target: NOTIFY wake (≤250ms) + fence + one stream read + (mention: zero / ambient: one cheap model call), then streaming. And long replies stop dying at the worker resource limit — a “natural feeling” killer worse than raw latency.

Local dev topology

bun run dev starts Vite through Portless + the same dispatcher daemon, pointed at the shared prod DB with DISPATCH_QUEUE=dispatch-dev-<user>; the local entries route enqueues to that queue (queue param on director_spawn). Deployed www keeps enqueuing to dispatch. Prod never depends on a laptop; the prod conductor structurally never sees dev tasks. If one thread gets triggers from both surfaces, the execution fence arbitrates one driver at a time and the beat row is shared truth. Dev daemon down → dev posts visibly queue, and the dev entries route warns when the dev queue has no live worker heartbeat — honest failure instead of a silently different pipeline.

The arbe-9801 questions, answered

  1. Origin stamping — superseded. Per-developer queue name (queue param on director_spawn), not an origin column or claim filter.
  2. Two claimers, one queue — never happens: distinct queues per surface. No filtered claims, no double-fire window, no adoption. Stale dev tasks sit on the dev queue and are never adopted by prod (local code was the point); the heartbeat warning makes them visible.
  3. Secrets — the daemon (prod and dev alike) holds DATABASE_URL, LLM keys, service-role Supabase, streams secret, JWT secret. The BACKSTAGE_SECRET www seam shrinks toward deletion as the HTTP drives disappear. The “thin conductor” principle is retired deliberately: it looked like hygiene but caused the held-open POST, the CF resource deaths, and the dev split. Recorded here as a decision, not drift.
  4. Celld — deleted, not replicated locally. The deterministic cell-down fallback (a second decision variant) dies with it. Mutual exclusion (fence) + a beat row + Absurd redelivery is the per-thread brain; the missing ingredient was co-locating decision with execution, not the cell.
  5. Blast radius — a dev daemon claims only its own queue, which only that developer’s www enqueues to: local code can only run threads you posted to locally. Data is still prod by construction (there is no local DB); the fence bounds cross-surface races.
  6. DX — daemon inside bun run dev by default; missing secrets fail loudly at daemon boot.

Rejected

  • Smarter inline path (real speaker decision in-process in www, dev-only or as the fallback) — a smarter second pipeline is still a second pipeline.
  • Origin-stamped tasks + filtered claims on one queue — per-dev queues get the same isolation with no filter logic and no orphan-adoption question.
  • Local celld replica — replicates the topology this decision deletes; cell state authority across laptop/prod is unsolvable by construction.
  • Prod adoption of stale dev tasks — running dev-posted turns on prod code defeats the purpose; fail visibly instead.
  • Advisory locks replacing the fence — the fence RPCs already exist, already outlive the cell, and are sufficient alone; no rewrite.
  • (Still standing from reply-turn-runtime.md: CF Queues, Durable Objects, hand-rolled claim tables.)

Risks

  • Restart behavior regresses slightly: daemon restart aborts in-flight turns and leans on redelivery + the prior-reply idempotency guard — bounded drain exists; acceptable at alpha.
  • The beat state machine ports from celld’s actor serialization to lock-under-fence — the one place new correctness bugs can enter; step 2’s focused test suite is half that task.
  • Daemon trust surface grows (all the secrets) — accepted above.
  • Turns occupy daemon slots (IO-bound, concurrency env-tunable) — fine at current scale.

Migration order

  1. arbe-228d — turn execution into the daemon (kills mid-stream deaths; cell protocol untouched).
  2. arbe-a16f — speaker decision in-process; celld deleted.
  3. arbe-e1f9 — dev parity: per-dev queue, daemon in bun run dev, inline path deleted. Ambient works locally here.
  4. arbe-448b — post_to_thread + /api/wf/step ride the same enqueue.