# Where in-process reply turns run Decision for arbe-8e14 (2026-07-07). Status: decided, implementation in flight. ## Goal **A thread never goes silent.** Any message to any agent gets a visible "working" state within seconds, and exactly one visible ending — the reply, or a failure that says why — even when the model thinks for minutes. Turn duration is bounded by our turn cap, never by infrastructure. The acceptance drill (glm-5.2 on openrouter as the adversarially slow model): 1. Slow happy path: new thread, standard agent, ask for a long generation. 201 immediately; turn visibly running; reply lands after >30s wall; exactly one usage row. (The arbe-8e14 repro, promoted to regression test.) 2. Visible failure: a failing turn (e.g. bogus model id) shows `dispatch.failed` with a readable reason within seconds. 3. Crash survival: restart the backstage mid-turn; the turn is redelivered and still ends exactly once. 4. Fast path unharmed: haiku on a short question stays at today's latency (queue hop adds ~a second). ## Problem In-process chat reply turns are handed to `waitUntil()` in `apps/www/src/routes/api/threads/[id]/entries/+server.ts` after the POST returns 201. Cloudflare cancels `waitUntil` work ~30s after the response, and cancellation is not a throw — so a slow turn (slow model, long generation) strands: no `pi.assistant`, no terminal signal, no usage row. Reproduced in thread qvnoqtpyqwtr. ## Decision **A reply turn is an Absurd task.** We already run the exact architecture this needs: Absurd (Postgres-native durable execution) driven by the backstage daemon on Fly (`arbe-backstage`, always-on, outbound-only). Shape, mirroring the existing `/api/wf/step` seam: 1. `POST /entries` spawns an Absurd task (`reply-turn`, queue `dispatch` — its own queue so chat never waits behind workflow steps) instead of calling `waitUntil`. Poster still gets 201 immediately. 2. The conductor claims it and stays thin: one held-open POST to `/api/dispatch/run` on www (behind `BACKSTAGE_SECRET`, same auth seam as `/api/wf/step`). A Worker has no wall-clock limit while the client holds the connection — the 30s cap is post-response `waitUntil` only. LLM keys and `@arbe/core` stay solely in www. 3. The endpoint executes the turn during the request and returns when the terminal signal is written. It starts with an idempotency guard: terminal signal already present → skip and 200. That makes Absurd's at-least-once redelivery (lease expiry, bounded retry) the self-heal for crashes. Parameters that must line up: - Turn cap: the endpoint aborts at ~10 min and writes `signal.dispatch.failed`. - Queue `claimTimeout` is 120s: the conductor heartbeats in-flight directed turns (`ctx.heartbeat(120)` every 30s, arbe-796a), so the lease stays fresh for long turns while an interrupted one frees its claim in ≤2 min instead of waiting out a giant timeout. - The conductor's request timeout for this handler sits between the two (~12 min); 4xx = permanent failure, 5xx/network = throw for Absurd's retry, same split as `driveStep`. Sequencing: the seam (endpoint + handler + migration creating the queue/spawn RPC) lands inert first; the producer flip in `/entries` (and any other `waitUntil(dispatch)` call site — `/api/wf/step` dispatches the same way, so workflow step turns share the strand class and the fix) follows once the migration is applied. Interim stopgap regardless: a ~25s deadline race on the `waitUntil` path that writes `signal.dispatch.failed` so a stall is visible and reclaimable. ## Rejected - **Cloudflare Queues** — new platform concept (binding, custom SvelteKit worker entry, DLQ config), 15-min cap, buys redelivery Absurd already gives us. - **Hand-rolled Postgres claim table (SKIP LOCKED + lease)** — Absurd *is* this, already installed, already operated. - **Streaming the reply on the open POST** — the Worker only lives while the poster stays connected; wrong reliability model for "201 now, reply lands on the durable stream later". - **Durable Objects / Cloudflare Workflows** — compute+state or step-shaped durable execution we don't need; state already lives in the durable stream. - **Hetzner VPS runner** — viable someday-home for the daemon (it is just a process), but the Fly conductor already exists; no new host. The sandbox/pi coding path (child thread) is unchanged — this decision covers in-process replies only. ## Drill runbook Prerequisites: both deploys live (conductor `fly deploy --ha=false`, then www). Task/queue inspection SQL and conductor log tailing → docs/system/ops/debugging.md ("Reply turns are Absurd tasks"). Setup — do NOT drop messages into an existing shared house (a bare house thread gets `filtered`: no bot is a thread member, and mention-gating on stale fixture agents is unreliable). Make a deterministic target instead: ```sh arbe agent create drillbot --house --trigger always --json # plain bot, no prompt straitjacket arbe thread create [-m ] # 1:1 thread under the bot — always dispatches ``` Timing: `arbe thread entries create` returning IS the 201; `arbe thread entries read` blocks until the dispatch terminal — wrap the pair in shell timestamps. 1. Slow happy path (measured 2026-07-07: 201 in 0.7s, `started` +4.4s, reply + `completed` +61s, 1 usage row): ```sh arbe thread create -m openrouter/z-ai/glm-5.2 arbe thread entries create "Write a ~1500-word story about a lighthouse keeper. Full story, no summary." arbe thread entries read # expect signal.dispatch.started within seconds, reply + signal.dispatch.completed after >30s wall cd packages && bunx supabase db query --linked "select count(*) from public.usage_events where thread_id = '' and seam = 'reply'" # exactly 1 row ``` 2. Visible failure (this IS the bogus-model test — `resolveReply` throws on an unknown slug, `withDispatchSignals` narrates; measured: failed terminal in 5.5s, task completes attempt 1, no retry): ```sh arbe thread create -m openrouter/nope/does-not-exist arbe thread entries create "hi" arbe thread entries read # signal.dispatch.failed: Unknown reply model "..." within seconds ``` 3. Crash survival: repeat drill 1's post, then mid-turn (after `signal.dispatch.started`, before the reply): ```sh fly apps restart arbe-backstage arbe thread entries read ``` Measured: the Worker finishes the held request even though its client died — the reply and terminal landed on time (+71s). The orphaned claim then waited out the 15-min lease; redelivery hit the entryId idempotency guard (run 2 result `{"outcome": "skipped"}`, task completed attempts=2) — one reply, one terminal, ever. All four drills passed against prod 2026-07-07. 4. Fast path unharmed (measured: 7.7s pre-flip → 8.2s post-flip; the queue hop costs ~0.5s): ```sh arbe thread create # no override → default openrouter/z-ai/glm-5.3-flash arbe thread entries create "What is 2+2?" arbe thread entries read ```