Skip to content
View as .md

Where in-process reply turns run

Decision for arbe-8e14 (2026-07-07). Status: decided, implementation in flight.

Goal

A thread never goes silent. Any message to any agent gets a visible “working” state within seconds, and exactly one visible ending — the reply, or a failure that says why — even when the model thinks for minutes. Turn duration is bounded by our turn cap, never by infrastructure.

The acceptance drill (glm-5.2 on openrouter as the adversarially slow model):

  1. Slow happy path: new thread, standard agent, ask for a long generation. 201 immediately; turn visibly running; reply lands after >30s wall; exactly one usage row. (The arbe-8e14 repro, promoted to regression test.)
  2. Visible failure: a failing turn (e.g. bogus model id) shows dispatch.failed with a readable reason within seconds.
  3. Crash survival: restart the backstage mid-turn; the turn is redelivered and still ends exactly once.
  4. Fast path unharmed: haiku on a short question stays at today’s latency (queue hop adds ~a second).

Problem

In-process chat reply turns are handed to waitUntil() in apps/www/src/routes/api/threads/[id]/entries/+server.ts after the POST returns 201. Cloudflare cancels waitUntil work ~30s after the response, and cancellation is not a throw — so a slow turn (slow model, long generation) strands: no pi.assistant, no terminal signal, no usage row. Reproduced in thread qvnoqtpyqwtr.

Decision

A reply turn is an Absurd task. We already run the exact architecture this needs: Absurd (Postgres-native durable execution) driven by the backstage daemon on Fly (arbe-backstage, always-on, outbound-only).

Shape, mirroring the existing /api/wf/step seam:

  1. POST /entries spawns an Absurd task (reply-turn, queue dispatch — its own queue so chat never waits behind workflow steps) instead of calling waitUntil. Poster still gets 201 immediately.
  2. The conductor claims it and stays thin: one held-open POST to /api/dispatch/run on www (behind BACKSTAGE_SECRET, same auth seam as /api/wf/step). A Worker has no wall-clock limit while the client holds the connection — the 30s cap is post-response waitUntil only. LLM keys and @arbe/core stay solely in www.
  3. The endpoint executes the turn during the request and returns when the terminal signal is written. It starts with an idempotency guard: terminal signal already present → skip and 200. That makes Absurd’s at-least-once redelivery (lease expiry, bounded retry) the self-heal for crashes.

Parameters that must line up:

  • Turn cap: the endpoint aborts at ~10 min and writes signal.dispatch.failed.
  • Queue claimTimeout is 120s: the conductor heartbeats in-flight directed turns (ctx.heartbeat(120) every 30s, arbe-796a), so the lease stays fresh for long turns while an interrupted one frees its claim in ≤2 min instead of waiting out a giant timeout.
  • The conductor’s request timeout for this handler sits between the two (~12 min); 4xx = permanent failure, 5xx/network = throw for Absurd’s retry, same split as driveStep.

Sequencing: the seam (endpoint + handler + migration creating the queue/spawn RPC) lands inert first; the producer flip in /entries (and any other waitUntil(dispatch) call site — /api/wf/step dispatches the same way, so workflow step turns share the strand class and the fix) follows once the migration is applied. Interim stopgap regardless: a ~25s deadline race on the waitUntil path that writes signal.dispatch.failed so a stall is visible and reclaimable.

Rejected

  • Cloudflare Queues — new platform concept (binding, custom SvelteKit worker entry, DLQ config), 15-min cap, buys redelivery Absurd already gives us.
  • Hand-rolled Postgres claim table (SKIP LOCKED + lease) — Absurd is this, already installed, already operated.
  • Streaming the reply on the open POST — the Worker only lives while the poster stays connected; wrong reliability model for “201 now, reply lands on the durable stream later”.
  • Durable Objects / Cloudflare Workflows — compute+state or step-shaped durable execution we don’t need; state already lives in the durable stream.
  • Hetzner VPS runner — viable someday-home for the daemon (it is just a process), but the Fly conductor already exists; no new host.

The sandbox/pi coding path (child thread) is unchanged — this decision covers in-process replies only.

Drill runbook

Prerequisites: both deploys live (conductor fly deploy --ha=false, then www). Task/queue inspection SQL and conductor log tailing → docs/system/ops/debugging.md (“Reply turns are Absurd tasks”).

Setup — do NOT drop messages into an existing shared house (a bare house thread gets filtered: no bot is a thread member, and mention-gating on stale fixture agents is unreliable). Make a deterministic target instead:

Terminal window
arbe agent create drillbot --house <fresh-or-own-house> --trigger always --json   # plain bot, no prompt straitjacket
arbe thread create <drillbot-id> [-m <model>]   # 1:1 thread under the bot — always dispatches

Timing: arbe thread entries create returning IS the 201; arbe thread entries read blocks until the dispatch terminal — wrap the pair in shell timestamps.

  1. Slow happy path (measured 2026-07-07: 201 in 0.7s, started +4.4s, reply + completed +61s, 1 usage row):
Terminal window
arbe thread create <drillbot-id> -m openrouter/z-ai/glm-5.2
arbe thread entries create <thread> "Write a ~1500-word story about a lighthouse keeper. Full story, no summary."
arbe thread entries read <thread>   # expect signal.dispatch.started within seconds, reply + signal.dispatch.completed after >30s wall
cd packages && bunx supabase db query --linked "select count(*) from public.usage_events where thread_id = '<thread>' and seam = 'reply'"   # exactly 1 row
  1. Visible failure (this IS the bogus-model test — resolveReply throws on an unknown slug, withDispatchSignals narrates; measured: failed terminal in 5.5s, task completes attempt 1, no retry):
Terminal window
arbe thread create <drillbot-id> -m openrouter/nope/does-not-exist
arbe thread entries create <thread> "hi"
arbe thread entries read <thread>   # signal.dispatch.failed: Unknown reply model "..." within seconds
  1. Crash survival: repeat drill 1’s post, then mid-turn (after signal.dispatch.started, before the reply):
Terminal window
fly apps restart arbe-backstage
arbe thread entries read <thread>

Measured: the Worker finishes the held request even though its client died — the reply and terminal landed on time (+71s). The orphaned claim then waited out the 15-min lease; redelivery hit the entryId idempotency guard (run 2 result {"outcome": "skipped"}, task completed attempts=2) — one reply, one terminal, ever. All four drills passed against prod 2026-07-07.

  1. Fast path unharmed (measured: 7.7s pre-flip → 8.2s post-flip; the queue hop costs ~0.5s):
Terminal window
arbe thread create <drillbot-id>   # no override → default openrouter/z-ai/glm-5.3-flash
arbe thread entries create <thread> "What is 2+2?"
arbe thread entries read <thread>