Debugging

Start with the smallest probe that can separate the likely causes. Houses, threads, and streams usually go through the regular CLI or HTTP surface; arbe debug is reserved for checks that bypass permissions or environment resolution. Scripted proofs live in tests/README.md.

If local Vite rejects a package export that is already present in that package’s package.json, inspect the actual target before changing code. The director notice proof hit stale export resolution after a new core schema was added; a human restart of the shared dev server resolved it. Passing typechecks does not prove a running resolver has refreshed. Do not delete caches, alter dependencies, or restart a shared server without its owner’s approval.

To see which version of a dependency is installed and who pulls it in, run bun why <package>; it’s shorter than reading bun.lock. bun’s package cache under node_modules/.bun keeps old versions nothing links to, so a version found by listing that folder proves nothing. A catalog bump only takes effect after bun install.

Testing layers

Only unit tests run offline.

Layer	Run	Needs
Unit / integration	`bun run test` (scope with `--filter '@arbe/<pkg>'`)	nothing
HTTP CRUD contract	`bun run tests/http-crud-proof.ts`	dev server + CLI auth
Env-bound dispatch	`bun run scripts/remote-dispatch.ts [<env>] --house <id> [--local]`	environment + sandbox + house bot
LLM prompt suite	hand an agent `tests/README.md`	running stack
Director scenarios	`bun tests/run-scenario.ts <n…> —house <id	new>` → run-scenarios skill
Browser / UI	`agent-browser`	dev server + browser profile
Task TUI layout	`bun run apps/cli/src/task-tui/dev/frames.tsx 120,99,70,50 24`	nothing

frames.tsx renders the task TUI headlessly and prints each frame as text, so you can check responsive breakpoints, truncation and modal clipping without a real terminal — pass --keys=l,? to rotate the pane layout or open a modal first. Markdown bodies stay blank in those frames; the headless renderer does not paint them.

When website behavior is genuinely in doubt, prove it in the website rather than substituting a CLI result — but a browser pass is not owed to every visible change; see the skill’s “Visible change proof”. Browser profile setup and login recovery live only in the browser-testing skill.

First probes

arbe --local debug env       # resolved URLs and LOCAL/REMOTE scope
arbe --local whoami     # token validates against the resolved backend
arbe http /api/me            # raw API; supports --jq, --status, -v, and stdin bodies
arbe env diagnose <env>      # dispatch readiness: capability, keys, secrets, sandbox
arbe thread diagnose <id>    # examine every director turn in the window; 2=any failed for good, 3=latest stalled
arbe thread trace <ref>      # trigger, pickup, turn phases, spend, terminal outcome
arbe thread entries list <id> --follow
arbe thread entries list <id>          # print a thread's messages (snapshot) — the reader
arbe thread entries read <id>          # waits for a reply in flight, exits when the dispatch settles — not a reader
durable-stream read arbe-thread-<id> --offset -1 --live

arbe http, thread commands, and stream proxy reads use the signed-in user.

Testing as a bot identity

ARBE_CONFIG_DIR gives each tester its own CLI identity, so multiple agents (or one agent playing two roles) can act in the same house without sharing one token:

arbe agent key <bot> --name tester          # mint a key for the bot
ARBE_CONFIG_DIR=~/.config/arbe-<bot> arbe login --token arbe_…
ARBE_CONFIG_DIR=~/.config/arbe-<bot> arbe house select <house-id>

ARBE_TOKEN=arbe_… arbe … is the lighter form for one command: it overrides the stored login without writing it, so no config dir or house select is needed (pass --house).

Every prefixed command posts and reads as that bot. Note the permission asymmetry this exposes: a human member legitimately sees every house they belong to, a bot key sees only what the bot is a member of — so when probing cross-house access, check auth whoami first or you will attribute your own human reach to the bot. Raw durable-stream reads use DURABLE_STREAMS_SECRET from .env.local (bunx varlock reveal DURABLE_STREAMS_SECRET) and bypass the app’s permission checks.

Fast discriminators:

Connection refused or curl HTTP 000 on localhost → distinguish a stopped server from the agent harness blocking network access. If portless doctor is healthy or the URL works in the human’s browser, retry the same read-only request (or packages/skills/verify-arbe/doctor.sh) through the harness’s normal network approval mechanism before declaring the server blocked. A successful approved request identifies a sandbox restriction; do not restart the shared dev server.
401 from whoami → run arbe login.
Stream-service 401 after switching to an older PR → compare its environment loader with main before rotating credentials. Pre-Varlock branches use $env/static/private instead of the shared schema imports. Update the PR’s base and restart dev before retrying; don’t copy secrets backward to support the old loader.
Local www with prod Supabase/Electric/streams → writes still hit production. arbe --local debug env warns about this split.
Installed CLI disagrees with the checkout → arbe --version shows its baked commit. Run from source (bun apps/cli/src/cli.ts …) or rebuild with arbe upgrade while proving a CLI fix.
A form submission returns the Cross-site … form submissions are forbidden body → CSRF, not authentication. apps/www/src/hooks.server.ts owns the guard; bearer requests are its deliberate exemption.

Test humans for scenarios

Director scenarios post as three real accounts (Dana, Eli, Fay) so the director sees different people. Mint, check and the AgentMail setup: the “Distinct humans” section of packages/skills/run-scenarios/SKILL.md. Sign-in mail hits a project-wide Supabase cap; a 429 over_email_send_rate_limit clears with time, not retries.

Verify a deployment

Check each runtime changed by the commit:

curl -fsS -H 'Cache-Control: no-cache' https://arbe.0sk.ar/api/version
fly logs -a arbe-backstage --no-tail | tail

arbe thread director (bare) ends with a backstage: line naming every worker that heartbeated in the last three minutes, or a ⚠ backstage offline warning. Read it before blaming a slow bot: a thread behind with no lease held and no beating backstage is a dead runner. Local bun run dev starts no worker: a healthy local HTTP preview cannot rescue a stopped production backstage.

The backstage has no HTTP service, so flyctl has no health check and will call a machine that came up stopped “in a good state” and exit 0. CI therefore runs verify-deploy after every deploy (every machine must be started), and the deploy job is serialized with a concurrency group: two pushes minutes apart once ran two deploys on top of each other, the second replaced the machine and left it stopped, and CI was green for 90 minutes of silence (2026-09-03). fly status -a arbe-backstage shows the machine state; fly machine start <id> -a arbe-backstage is the remedy.

/api/version identifies www only. The backstage version appears in its startup logs. Dispatch turns from packages/core/dispatch/ execute inside the backstage, so a core dispatch change ships with the backstage, not with www. A cross-package change may require both deployments.

After deploy, post a fresh entry, then inspect arbe thread trace <thread> and backstage logs. Static assets can remain edge-cached briefly; retry with Cache-Control: no-cache and Pragma: no-cache before concluding the new asset did not ship.

Workers Build failed

A red Workers Builds: arbe (or arbe-docs) check on a commit says nothing about why. Read the build log with Cloudflare’s cf CLI, not the dashboard and not wrangler, which has no builds command: cf builds list for the build uuid, then cf builds logs get <uuid>. The exact commands and Worker tags are in deployment.

Backstage image fails to build

The Deploy backstage step fails with Unknown lockfile version while www and docs build green: the image’s bun is older than whatever wrote bun.lock. Keep FROM oven/bun:<version>-slim at or ahead of the bun that rewrites the lockfile (a local bun install on a newer bun upgrades it silently). www and docs going green hides the fallout — bots, workflows and every dispatch turn run in the backstage, so read the last green Build with gh run list before believing a dispatch change is live.

The docs are served at https://arbe.0sk.ar/docs through www’s DOCS service binding, so they need both workers. Deploy docs before www — scripts/deploy.ts already orders them that way — and verify with curl -sI https://arbe.0sk.ar/docs/system/architecture/ plus curl -s https://arbe.0sk.ar/llms.txt | head -1.

Docs site builds stale

astro build caches per-package in apps/docs/node_modules/.astro, and that cache survives a dist/ wipe. An edit to astro.config.mjs — a remark plugin, base, site — can therefore appear to do nothing while the build reports success. Symptom: some pages carry the new output and some carry the old. Clear both caches before concluding the config is wrong:

rm -rf apps/docs/.astro apps/docs/node_modules/.astro apps/docs/dist
bun run --filter '@arbe/www-docs' build

svelte-check reports a file that does not exist

svelte-check --tsgo --incremental transpiles the Svelte tree to disk under apps/www/.svelte-kit/.svelte-check/ and reuses it. Change apps/www/tsconfig.json — especially files/include — and the overlay tsconfig written next to that cache keeps the old shape, so the check fails with Error: File '.../.svelte-check/svelte/<name>.d.ts' not found for a path nothing references any more. The source is correct; the cache is not.

rm -rf apps/www/.svelte-kit
bun run --filter '@arbe/www' check

`X is not defined` only in the browser

@arbe/core is shared by the server, the CLI, and the browser, and nothing marks which modules are browser-safe. Buffer, process, and node:* imports type-check fine (globally typed) and pass bun run test (bun has them), so they only fail once a real browser loads the page. Before importing a core module from browser code, check it for Node-only APIs; use web-standard ones (TextEncoder, crypto.subtle, Uint8Array) or keep the code behind a lib/server or .server.ts boundary.

A `redirect()` in a page load 500s on a cold load

apps/www/src/routes/+layout.ts sets ssr = false, so every +page.ts/+layout.ts load runs in the browser. A redirect() thrown from a page load on the first render of a URL does not navigate: the Redirect reaches handleError instead, the analytics console logs client exception: {status: 307, location: …}, and the visitor gets the 500 page. Redirect from the component with goto(href, { replaceState: true }) in an $effect, or from hooks.server.ts, not from a page load.

While you are there: deleting a route file Vite is currently serving crashes bun run dev with an ENOENT for the deleted path. Write the replacement first, delete second, and expect to restart the dev server if you get the order wrong.

CLI output truncates at 65536 bytes

bun buffers stdio into its node-stream shim, and process.exit throws away whatever the 64KiB pipe buffer has not swallowed — so a piped CLI command stops at exactly 65536 bytes (arbe-7ef2, arbe-cbd4). Three things make this hard to see, all measured on bun 1.3.14:

console.log is not exempt. It writes straight to the fd in a bare script, but is buffered like everything else once anything has touched process.stdout — which the CLI does on every run. It also never calls the stream’s write method, so a process.stdout.write shim never sees it.
Bun.spawn({ stdout: 'pipe' }) does not reproduce it. It returns the full bytes even from a truncating binary; the first fix for arbe-cbd4 shipped green and broken on such a test. Reproduce and assert through a real shell pipe: sh -c 'bun cli … | cat | wc -c'.
writableLength / needDrain read 0 right after a 70KB write that returned false, so any “is something pending?” gate never fires.

The fix is installSyncStdio() (apps/cli/src/output.ts), called once at the CLI entry: it routes both the streams and the console methods through a blocking write(2). Two traps if you touch it — bun leaves piped stdio non-blocking, so a full pipe raises EAGAIN (retry, don’t let it throw), and blocking defers SIGINT until the reader drains.

Website performance

Slow page opening or thread switching → measurement recipe. Start with bun run apps/www/scripts/profile-routes.ts --help; the profiler is read-only.

Keep the browser, fixtures, and source quiet during each run. Save JSON, compare with apps/www/scripts/compare-performance.ts, and repeat both versions before claiming a gain. Local Vite measurements do not establish cold-start or production performance.

Baseline, botlab fixtures, and evidence — including a repeat that disproved an apparent speedup.

Docs font flicker

The docs dev server (astro dev) re-fetches fonts on every page, so text flickers on each click there. That is dev-only; judge font loading against the production build: cd apps/docs && bun run build && bunx wrangler dev --port 8790, then open http://localhost:8790/docs/.

Thread creation

POST /api/threads answers with a Server-Timing header naming every remote wait it did — scope (membership gate), parent (what createThread waited on for the resolved parent; ≈scope because the gate is handed over in flight), stream (durable-stream create), then insert / config / participants. Read it from curl, the browser network panel, or the bench harness:

cd apps/www
bun run scripts/bench-thread-create.ts --house <house_id> --bot <agent_uuid>
bun run scripts/bench-thread-create.ts --house <id> --bot <uuid> --runs 7 --json
bun run scripts/bench-thread-create.ts --origin https://arbe.0sk.ar --house <id> --bot <uuid>

Use a disposable house — every run creates a real thread, and they are deleted afterwards unless --keep. Three shapes are measured: house (house parent, pregenerated id, seeded participant), dm (agent parent), and onboarding (bot, membership, config-seeded interview thread, and greeting). Onboarding bots are deleted after their threads, so repeated benchmarks do not consume the house spawn cap.

What the numbers said in Aug 2026 (arbe-76f9): each Postgres round trip costs 45–75 ms, and durable-stream create costs 450–1000 ms — more than everything else combined, on localhost and in production alike (raw PUT to Electric measures the same, so it is the provider’s cost, not ours). Anything that only reorders our own Postgres trips is therefore worth ~100-200 ms at most; the stream is the floor while thread creation waits on it (arbe-5660).

Repair localhost TLS

Run portless doctor first. bun run dev:trust retries Portless’s local certificate-authority trust; it does not repair an unhealthy proxy or route. The proxy must listen on 443, which needs root; on a dev machine it runs as a boot service (portless service install, once, by the human). If bun run dev says the proxy isn’t running and no TTY is available for sudo, that is a blocker: say so and ask the human to run portless service install. Never take Portless’s suggested -p 1355 fallback, because Portless remembers that port and every later start serves https://arbe.localhost:1355 instead. A foreign/shared bun run dev process is a blocker: ask its owner rather than stopping or restarting it.

Open the dev server from another device

To reach local dev from a phone or laptop on the tailnet, serve Vite’s direct port on a spare HTTPS port: sudo tailscale serve --bg --yes --https=8445 http://localhost:8888, then open https://<node>.<tailnet>.ts.net:8445. Vite listens on localhost, so use that rather than 127.0.0.1. vite.config.ts already allows .ts.net hosts. Check tailscale serve status first because other ports may already be taken, and stop the share with sudo tailscale serve --https=8445 off. Use serve, not funnel: funnel makes the share public.

Dispatch

The diagnostic pipeline is:

conversation entry → wake_thread on configured queue → claim thread lease
                   → read past decided_offset → decision → selected turn or nobody
                   → tagged outcome → advance offset → catch up again

The mechanism is documented in dispatch and thread director. Debug from the durable evidence:

arbe thread diagnose <thread>
arbe thread trace <thread>
arbe thread entries list <thread> --json
# lease renewals, catch-up passes, decisions, and turn internals are backstage logs:
fly logs -a arbe-backstage --no-tail | tail

Picks and silence over new conversation write signal.thread_director.decision, whether chosen by rules or a model; a bot attempt then writes a tagged signal.thread_director.outcome. Bookkeeping-only windows advance decided_offset without another decision. Current selected-turn skip reasons are:

Reason	Meaning
`filtered`	the selected turn produced nothing more specific
`budget_exceeded`	the house spend gate refused the selected turn
`empty_reply`	the model completed without usable final text

arbe thread director <ref> distinguishes caught-up, behind, and live-lease states. arbe thread trace <ref> shows persisted decisions, outcomes, dispatch signals, and historical signal types. Reader copy for selected-turn failures and skips lives in reader-copy.ts.

Presence cues

Replay the durable signals that drive queued/thinking/working state:

arbe thread entries list <thread> --json | \
  (cd apps/www && bun run scripts/replay-presence.ts)

The pre-signal “Deciding who replies…” and its stall message are client state, owned by composer-cues.ts. A stall reports a browser $exception because no terminal or progress signal arrived after a durable post.

Dispatch latency

Start with arbe thread trace <thread> --house <house>. Each completed bot reports total trigger-to-completion time separately from its own turn; the outer runner is bookkeeping, not another reply. An explicit trigger entry id wins over later messages, so a human posting during generation cannot make the reply appear faster. Older inferred anchors say estimated; missing referenced entries stay unknown. Handoff is part of pickup, not an additional delay to add.

New director decisions record the effective human-burst pause and the measured decision duration. One participant bot defaults to no pause; explicit scope settings win, and multi-bot threads retain the app default. This pause is not measured sleeping time: worker pickup and reads can overlap that interval. New turns report every model call’s first nonempty text/thinking/tool delta and first text delta, measured from that call’s start; these are provider-stream observations, not the moment text appeared in the browser. No delta means not observed, while old turns say the timing was not recorded.

Use the trace’s PostHog link for the same turn’s usage analytics. System-prompt and tool-schema sizes are characters, not token estimates; message/tool counts explain why a clean conversation still has a large input. PostHog receives these counts and timings, never the exact prompt, answer, or request body. Historical exact context cannot be reconstructed from analytics. Check existing telemetry before proposing another capture store.

The busy bar follows durable prepare/generate/tools/save transitions, with an indeterminate active stage. The initial choose stage includes the human-burst wait; it does not claim the director has started a model call. Progress writes are included in Arbe processing, outside measured provider and tool time.

arbe thread trace <ref> pairs each persisted signal.thread_director.decision with dispatch start/completion and its tagged outcome. It reports pickup, handoff, reply-turn phases, tokens, and cost when those fields exist. Phase fields do not sum to the full duration; use fresh production samples rather than old medians when investigating latency. New modelMs measures provider-call wall time and toolsMs measures tool execution; historical llmMs includes both plus transcript writes. The completed-line Arbe remainder excludes both, but begins at dispatch start: pickup and handoff still contain earlier Arbe work and intentional debounce. Compare no-tool turns with the same model and record the debounce setting; do not treat a shorter configured pause as faster processing.

Before judging a scenario’s silence or speaker selection, verify each bot’s effective thread mode. Creating a thread with one bot seeds an always override even if that bot is ambient at house scope. The scenario runner must set and read back its declared modes on every new thread before posting; scenario 19 otherwise answers “Perfect” correctly for its actual configuration while failing the ambient expectation.

For a workflow instruction followed by no eligible bots, inspect the recorded author kind before changing prompts or adding mentions. Workflow instructions are authored by the house System agent and the assigned bot has an always override; resolving System as unknown prevents that bot from owing a reply.

Prompt caching defaults

Bot replies build their context in packages/core/dispatch/dispatch.ts, then agent-turn.ts calls pi-ai’s stream/streamSimple from the Agent’s stream function. They request usage, pass the thread id as sessionId, request cacheRetention: 'long', and send OpenRouter’s explicit x-session-id header. The header pins a conversation even before its first cache hit and supplies Z.AI’s upstream session affinity; prompt_cache_key alone is OpenRouter’s fallback routing key. Neither option guarantees a hit or forces an unsupported provider to cache.

Defaults by upstream provider (OpenRouter reference, checked 2026-09-13):

Z.AI (including GLM-5.3-Flash) and DeepSeek (including V4 Flash): automatic, no cache_control needed. The existing read-only evidence in arbe thread trace tplxswluvwvu --house xkwkqzyovpwm includes GLM reads of 4,608 tokens and DeepSeek reads of 7,424, interspersed with misses. Zero writes does not mean caching is disabled.
OpenAI, xAI, Moonshot: automatic; OpenAI requires at least 1,024 prompt tokens. OpenAI GPT-5.6+ also offers explicit breakpoints, but Arbe leaves its automatic caching enabled. Groq’s automatic caching is model-specific (Kimi K2).
Anthropic: requires opt-in. pi-ai already marks the system prompt, last tool, and conversation tail. Arbe’s long requests a one-hour TTL where supported, rather than the default five minutes; writes cost more than ordinary input. Minimum cacheable length depends on the Claude model.
Alibaba: explicit five-minute markers on the supported unsuffixed model ids listed in dispatch/reply.ts; pi-ai marks the system, last tool and conversation tail. Snapshot endpoints are deliberately excluded. The policy applies to both bundled and dynamically resolved models, without changing routing.
Gemini 2.5+: implicit caching, with model-specific minimum sizes; Arbe does not request its optional explicit cache storage or pay its storage charge. Other models/endpoints have no blanket caching guarantee.

The reusable prompt starts with identity, platform and authored instructions; changeable thread title/tags/run state and per-turn directives follow them. No current clock or trace id is injected ahead of that prefix. Tool schemas have a stable order within a turn; changing enabled tools or reaching the forced final answer without tools changes the provider’s prefix. Sliding the conversation window also invalidates some cached history. Retention, eviction and routing can still cause misses.

Trace and usage ledger use pi-ai’s reported cached_tokens and cache_write_tokens, not estimates; ordinary input excludes both. A reported zero is not proof that the provider cannot cache. Do not infer a cache hit from cacheRetention or prompt size. After deploying backstage, send two nearby replies with the same bot and tools in a fresh test-house thread, then inspect arbe thread trace <thread> --house <test-house>: look for positive cache read, including later calls within a tool turn. Never post into the read-only evidence house above. The offline wire regression is dispatch/prompt-caching.test.ts; it verifies headers, Alibaba markers and usage mapping through the installed pi-ai adapter, not a simulated cache hit.

A turn is stalled or missing its terminal

thread diagnose returns the trigger entry and trace id. Correlate them with the durable task and its attempts:

# run from packages/
bunx supabase db query --linked "select task_id, state, attempts, max_attempts, enqueue_at from absurd.t_dispatch where task_name='wake' and params->>'threadId'='<thread>' order by enqueue_at desc" -o table

bunx supabase db query --linked "select run_id, attempt, state, claimed_by, claim_expires_at, failure_reason from absurd.r_dispatch where task_id='<task>' order by attempt" -o table

# The trace's cheap record lives in three real surfaces: `arbe thread trace <thread>`
# for per-turn LLM timings, `(cd apps/www && wrangler tail arbe)` for request-level
# logs from the www Worker, and the `usage_events` ledger (joined by trace_id) for
# spend. There is no `posthog-cli`; query the ledger directly:
bunx supabase db query --linked "select seam, key_source, model_ref, input_tokens, output_tokens, cost_usd, created_at from usage_events where trace_id='<trace>' order by created_at" -o table

Interpret the failure shape before changing a timeout:

Turn cap → the core runner aborts the selected turn and writes signal.dispatch.failed; the director records a failure outcome.
Paid decision failure → the wake retries when retryable; when the budget is spent, or the model names a bot that isn’t a candidate, the decision fails visibly (a failure outcome with the reason), never a silent nobody.
Lease loss → the stale pass stops before another append; another wake or the behind scan catches the thread up.
Lease contention → the second wake returns quietly; the pass holding the lease keeps reading until the thread is caught up, and the janitor is the backstop.
[thread-director.scan] stranded thread <id> → the five-minute janitor had to wake a thread, which means a wake was lost or a pass ended early. In a healthy system this line never appears; find out why that thread fell behind (arbe thread trace <id> shows the gap as a slow pickup marked ⚠).
[thread-director.pass] with repeated settlement:stay → inspect the tagged outcomes for that decision; three transient failures or one permanent failure should let the offset advance.
[thread-director.steer] → the director routed that many mid-reply messages into the running reply. Whether the reply took them shows as steered=<entry ids> on its outcome line in arbe thread trace (it appears only there, not where the message landed); a routed message missing there arrived after the reply’s last tool round and stays owed.
Backstage process loss → the task claim and thread lease expire, then a fresh claim reconstructs settlement from the stream. A reply in flight resumes in its own reply task about two minutes later, replaying what it recorded.
arbe thread director behind with no lease while a bot is “thinking” → normal: the pick’s reply task owns the turn and wakes the director when it settles. Inspect it, and the steps it recorded, by decision id:

bunx supabase db query --linked "select task_id, state, attempts from absurd.t_dispatch where task_name = 'reply' and params->>'decisionEntryId' = '<decision>'" -o table
bunx supabase db query --linked "select checkpoint_name, updated_at from absurd.c_dispatch where task_id = '<task>' order by updated_at" -o table

Turn caps live in packages/core/dispatch/reply-turn.ts; lease intervals live at the top of apps/backstage/src/thread-director-wake.ts; reply retries, claims and the per-attempt cap at the top of apps/backstage/src/reply-task.ts. A reply task counts its failures, and remembers its final append and its owed usage, in its own checkpoints (failure:N, final, bill:N), not on the stream.

Spend is joined by the turn’s trace id. usage_events is the ledger; PostHog is the second sink. See observability and analytics.

Env-bound dispatch

Separate “did dispatch run?” from “could the sandbox answer?”:

arbe env diagnose <env>
arbe thread diagnose <thread>
arbe sandbox view <id>       # live daytona state
arbe x -s <sandbox> -- bash -lc 'tail ~/arbe-pi-runner.log; tail ~/pi-run.log'

run_command results appear as pi.tool_result entries. For a complete proof, bun run scripts/remote-dispatch.ts [<env>] --house <id> [--local] plants a sandbox-only nonce and requires the bot to read it back. See the manual runbook and Daytona.

Detached work runs on a child thread. Start diagnosis there; snapshots from thread entries list are safer for agents than --follow, which intentionally never returns. A child with no terminal should be checked against the runner logs above, then against its task.

Coding runs

A coding run is one coding-run task in backstage; its id is the run_id on the child’s row. Read the row, then the task and its checkpoints (the absurd schema is not on PostgREST, so from packages/supabase):

bunx supabase db query --linked "select run_state, run_id, run_epoch, run_ending, run_wake from threads where id='<child>'" -o table
bunx supabase db query --linked "select state, attempts, completed_payload from absurd.t_default where task_id='<run_id>'" -o table
bunx supabase db query --linked "select checkpoint_name, state from absurd.c_default where task_id='<run_id>' order by updated_at" -o table

A coding run writes no “working” entry between steps, so entries list can look idle. signal.thread.pi_heartbeat entries (about every 5s, shown with --show-signals) mean the runner is alive and the run is working; their absence for minutes is the stall to chase.

claim, launch (holds error when the box or runner failed), end (the decision), fence, handover, close, next (the chat that started the next run) are written in that order; the last one present is where a stuck run stopped. A run parked between them shows sleeping with run_wake rising about once a minute while its runner beats. A row stuck at run_ending = true means the task is retrying the parent handover.

If a thread keeps reaching an old sandbox after rebinding environments, follow stale thread sandbox pointers.

Backstage

arbe wf runs heads its listing with the specific condition and its remedy — read it before reaching for a restart, because only two of the four cases are the backstage’s fault:

Header line	What it means	Remedy
runner offline	no heartbeat in three minutes	restart the backstage
beating but has not claimed N runs	runnable work sitting unclaimed — wedged	restart the backstage
parked with no await anchor	no `wf_run_threads` row, so nothing can emit the wake event	cancel and respawn; see arbe-4478
waiting on a person / a bot turn	parked on `ctx.awaitEvent`; a human gate waits as long as it takes, a bot turn past an hour is lost	open the run thread
`<task>` has retried N times	uncapped or pathological retry ladder	fix the task — a restart only makes it retry sooner

A high retried N× on a finished row is the same signal after the fact.

fly logs -a arbe-backstage --no-tail
bun run --filter '@arbe/backstage' deploy

Use the package deploy script. Bare fly deploy from apps/backstage has the wrong Docker build context. A post-deploy Cannot find module './x.js' crash means the partial image missed a transitive import; bun run check includes check:backstage-image, which identifies imports outside the Docker COPY set or missing production dependencies.

Director model experiments are process-wide backstage variants. Set both values explicitly, deploy/restart through the package script, then post a fresh botlab entry and read the durable proof:

fly secrets set -a arbe-backstage \
  ARBE_DIRECTOR_MODEL=openai/gpt-5.6-terra \
  ARBE_DIRECTOR_THINKING_LEVEL=high
bun run --filter '@arbe/backstage' deploy
fly logs -a arbe-backstage --no-tail | tail  # directorModel=… directorThinking=…
arbe thread trace <fresh-thread>                       # model=… thinking=…

ARBE_DIRECTOR_MODEL accepts either openai/gpt-5.6-terra or openrouter/openai/gpt-5.6-terra form and normalizes both to the canonical startup/trace ref. The default off sends no reasoning parameter, leaving reasoning behavior to the provider default. Unset both secrets to return to the code-owned defaults (openai/gpt-5.6-terra, off): fly secrets unset -a arbe-backstage ARBE_DIRECTOR_MODEL ARBE_DIRECTOR_THINKING_LEVEL, then deploy/restart and verify the same two surfaces. Blank values also act as unset. Startup fails before polling on malformed nonblank values, unknown models, or unsupported model/thinking combinations. Only paid decision witnesses show model=… thinking=…; model-free decisions say no llm time and omit the configured variant.

Production backstage on Fly consumes all background work, including work submitted through localhost. bun run dev runs only www/API and docs; apps/www/tmp/dev.log contains local HTTP logs. Read worker activity with fly logs -a arbe-backstage and durable decisions/outcomes with arbe thread trace <thread> --house <house>. Trace records the executing backstage SHA, including failures; unstamped older records say unknown. A later deployment does not change this attribution.

Check local HTTP/auth readiness separately from production worker heartbeat. Before proving changed background code, deploy it from main, then submit fresh work in a test house. Test houses isolate records, not deployment effects. An invalid queued payload must fail before execution; the worker log identifies its kind/id, validation details, running SHA, and deploy-first hint. A schema-valid payload can still depend on changed semantics, so a passing shape check does not replace coordinated deployment.

Every failed command, surprising flag, and source-only discovery is debugging-surface feedback. Fix the command, proof, skill, or canonical doc so the next investigation starts with the discriminator.