Skip to content
View as .md

Where in-process reply turns run

Decision for arbe-8e14 (2026-07-07). Status: decided, implementation in flight.

Goal

A thread never goes silent. Any message to any agent gets a visible “working” state within seconds, and exactly one visible ending — the reply, or a failure that says why — even when the model thinks for minutes. Turn duration is bounded by our turn cap, never by infrastructure.

The acceptance drill (glm-5.2 on openrouter as the adversarially slow model):

  1. Slow happy path: new thread, standard agent, ask for a long generation. 201 immediately; turn visibly running; reply lands after >30s wall; exactly one usage row. (The arbe-8e14 repro, promoted to regression test.)
  2. Visible failure: a failing turn (e.g. bogus model id) shows dispatch.failed with a readable reason within seconds.
  3. Crash survival: restart the backstage mid-turn; the turn is redelivered and still ends exactly once.
  4. Fast path unharmed: haiku on a short question stays at today’s latency (queue hop adds ~a second).

Problem

In-process chat reply turns are handed to waitUntil() in apps/www/src/routes/api/threads/[id]/entries/+server.ts after the POST returns 201. Cloudflare cancels waitUntil work ~30s after the response, and cancellation is not a throw — so a slow turn (slow model, long generation) strands: no pi.assistant, no terminal signal, no usage row. Reproduced in thread qvnoqtpyqwtr.

Decision

A reply turn is an Absurd task. We already run the exact architecture this needs: Absurd (Postgres-native durable execution) driven by the backstage daemon on Fly (arbe-backstage, always-on, outbound-only).

Shape, mirroring the existing /api/wf/step seam:

  1. POST /entries spawns an Absurd task (reply-turn, queue dispatch — its own queue so chat never waits behind workflow steps) instead of calling waitUntil. Poster still gets 201 immediately.
  2. The conductor claims it and stays thin: one held-open POST to /api/dispatch/run on www (behind BACKSTAGE_SECRET, same auth seam as /api/wf/step). A Worker has no wall-clock limit while the client holds the connection — the 30s cap is post-response waitUntil only. LLM keys and @arbe/core stay solely in www.
  3. The endpoint executes the turn during the request and returns when the terminal signal is written. It starts with an idempotency guard: terminal signal already present → skip and 200. That makes Absurd’s at-least-once redelivery (lease expiry, bounded retry) the self-heal for crashes.

Parameters that must line up:

  • Turn cap: the endpoint aborts at ~10 min and writes signal.dispatch.failed.
  • Queue claimTimeout is 120s: the conductor heartbeats in-flight directed turns (ctx.heartbeat(120) every 30s, arbe-796a), so the lease stays fresh for long turns while an interrupted one frees its claim in ≤2 min instead of waiting out a giant timeout.
  • The conductor’s request timeout for this handler sits between the two (~12 min); 4xx = permanent failure, 5xx/network = throw for Absurd’s retry, same split as driveStep.

Sequencing: the seam (endpoint + handler + migration creating the queue/spawn RPC) lands inert first; the producer flip in /entries (and any other waitUntil(dispatch) call site — /api/wf/step dispatches the same way, so workflow step turns share the strand class and the fix) follows once the migration is applied. Interim stopgap regardless: a ~25s deadline race on the waitUntil path that writes signal.dispatch.failed so a stall is visible and reclaimable.

Rejected

  • Cloudflare Queues — new platform concept (binding, custom SvelteKit worker entry, DLQ config), 15-min cap, buys redelivery Absurd already gives us.
  • Hand-rolled Postgres claim table (SKIP LOCKED + lease) — Absurd is this, already installed, already operated.
  • Streaming the reply on the open POST — the Worker only lives while the poster stays connected; wrong reliability model for “201 now, reply lands on the durable stream later”.
  • Durable Objects / Cloudflare Workflows — compute+state or step-shaped durable execution we don’t need; state already lives in the durable stream.
  • Hetzner VPS runner — viable someday-home for the daemon (it is just a process), but the Fly conductor already exists; no new host.

The sandbox/pi coding path (child thread) is unchanged — this decision covers in-process replies only.

Drill runbook

Prerequisites: both deploys live (conductor fly deploy --ha=false, then www). Task/queue inspection SQL and conductor log tailing → docs/system/ops/debugging.md (“Reply turns are Absurd tasks”).

Setup — do NOT drop messages into an existing shared house (a bare house thread gets filtered: no bot is a thread member, and mention-gating on stale fixture agents is unreliable). Make a deterministic target instead:

Terminal window
arbe agent create drillbot --house <fresh-or-own-house> --trigger always --json   # plain bot, no prompt straitjacket
arbe thread create <drillbot-id> [-m <model>]   # 1:1 thread under the bot — always dispatches

Timing: arbe thread entries create returning IS the 201; arbe thread entries read blocks until the dispatch terminal — wrap the pair in shell timestamps.

  1. Slow happy path (measured 2026-07-07: 201 in 0.7s, started +4.4s, reply + completed +61s, 1 usage row):
Terminal window
arbe thread create <drillbot-id> -m openrouter/z-ai/glm-5.2
arbe thread entries create <thread> "Write a ~1500-word story about a lighthouse keeper. Full story, no summary."
arbe thread entries read <thread>   # expect signal.dispatch.started within seconds, reply + signal.dispatch.completed after >30s wall
cd packages && bunx supabase db query --linked "select count(*) from public.usage_events where thread_id = '<thread>' and seam = 'reply'"   # exactly 1 row
  1. Visible failure (this IS the bogus-model test — resolveReply throws on an unknown slug, withDispatchSignals narrates; measured: failed terminal in 5.5s, task completes attempt 1, no retry):
Terminal window
arbe thread create <drillbot-id> -m openrouter/nope/does-not-exist
arbe thread entries create <thread> "hi"
arbe thread entries read <thread>   # signal.dispatch.failed: Unknown reply model "..." within seconds
  1. Crash survival: repeat drill 1’s post, then mid-turn (after signal.dispatch.started, before the reply):
Terminal window
fly apps restart arbe-backstage
arbe thread entries read <thread>

Measured: the Worker finishes the held request even though its client died — the reply and terminal landed on time (+71s). The orphaned claim then waited out the 15-min lease; redelivery hit the entryId idempotency guard (run 2 result {"outcome": "skipped"}, task completed attempts=2) — one reply, one terminal, ever. All four drills passed against prod 2026-07-07.

Superseded 2026-10-03: turns now run inside the backstage, so a restart kills the turn itself and there is no idempotency guard to hit. The redelivered wake task replays the turn’s recorded model calls and tool runs from its Absurd checkpoints, its appends dedupe by producer sequence, and the reply lands in one append with its director outcome. A tool that changes something and was cut off mid-run is not repeated; the bot is told it may or may not have run.

  1. Fast path unharmed (measured: 7.7s pre-flip → 8.2s post-flip; the queue hop costs ~0.5s):
Terminal window
arbe thread create <drillbot-id>   # no override → default openrouter/z-ai/glm-5.3-flash
arbe thread entries create <thread> "What is 2+2?"
arbe thread entries read <thread>