# Debugging Start with the smallest probe that can separate the likely causes. Houses, threads, and streams usually go through the regular CLI or HTTP surface; `arbe debug` is reserved for checks that bypass permissions or environment resolution. Scripted proofs live in [`tests/README.md`](../../../tests/README.md). If local Vite rejects a package export that is already present in that package’s `package.json`, inspect the actual target before changing code. The director notice proof hit stale export resolution after a new core schema was added; a human restart of the shared dev server resolved it. Passing typechecks does not prove a running resolver has refreshed. Do not delete caches, alter dependencies, or restart a shared server without its owner’s approval. ## Testing layers Only unit tests run offline. | Layer | Run | Needs | |---|---|---| | Unit / integration | `bun run test` (scope with `--filter '@arbe/'`) | nothing | | HTTP CRUD contract | `bun run tests/http-crud-proof.ts` | dev server + CLI auth | | Env-bound dispatch | `bun run scripts/remote-dispatch.ts [] [--local]` | environment + sandbox + house bot | | LLM prompt suite | hand an agent [`tests/README.md`](../../../tests/README.md) | running stack | | Director scenarios | `bun tests/run-scenario.ts --house ` → [run-scenarios skill](../../../packages/skills/run-scenarios/SKILL.md) | CLI auth + a fresh test house | | Browser / UI | [`agent-browser`](../../../packages/skills/browser-testing/SKILL.md) | dev server + browser profile | | Task TUI layout | `bun run apps/cli/src/task-tui/dev/frames.tsx 120,99,70,50 24` | nothing | `frames.tsx` renders the task TUI headlessly and prints each frame as text, so you can check responsive breakpoints, truncation and modal clipping without a real terminal — pass `--keys=l,?` to rotate the pane layout or open a modal first. Markdown bodies stay blank in those frames; the headless renderer does not paint them. When website behavior is genuinely in doubt, prove it in the website rather than substituting a CLI result — but a browser pass is not owed to every visible change; see the skill's "Visible change proof". Browser profile setup and login recovery live only in the [browser-testing skill](../../../packages/skills/browser-testing/SKILL.md). ## First probes ```sh arbe --local debug env # resolved URLs and LOCAL/REMOTE scope arbe --local whoami # token validates against the resolved backend arbe http /api/me # raw API; supports --jq, --status, -v, and stdin bodies arbe env diagnose # dispatch readiness: capability, keys, secrets, sandbox arbe thread diagnose # examine every director turn in the window; 2=any failed for good, 3=latest stalled arbe thread trace # trigger, pickup, turn phases, spend, terminal outcome arbe thread entries list --follow arbe thread entries list # print a thread's messages (snapshot) — the reader arbe thread entries read # waits for a reply in flight, exits when the dispatch settles — not a reader durable-stream read arbe-thread- --offset -1 --live ``` `arbe http`, thread commands, and stream proxy reads use the signed-in user. ### Testing as a bot identity `ARBE_CONFIG_DIR` gives each tester its own CLI identity, so multiple agents (or one agent playing two roles) can act in the same house without sharing one token: ```sh arbe agent key --name tester # mint a key for the bot ARBE_CONFIG_DIR=~/.config/arbe- arbe login --token arbe_… ARBE_CONFIG_DIR=~/.config/arbe- arbe house select ``` Every prefixed command posts and reads as that bot. Note the permission asymmetry this exposes: a human member legitimately sees every house they belong to, a bot key sees only what the bot is a member of — so when probing cross-house access, check `auth whoami` first or you will attribute your own human reach to the bot. Raw `durable-stream` reads use `DURABLE_STREAMS_SECRET` from `.env.local` (`bunx varlock reveal DURABLE_STREAMS_SECRET`) and bypass the app's permission checks. Fast discriminators: - Connection refused or curl HTTP `000` on localhost → distinguish a stopped server from the agent harness blocking network access. If `portless doctor` is healthy or the URL works in the human's browser, retry the same read-only request (or `packages/skills/verify-arbe/doctor.sh`) through the harness's normal network approval mechanism before declaring the server blocked. A successful approved request identifies a sandbox restriction; do not restart the shared dev server. - `401` from `whoami` → run `arbe login`. - Local www with prod Supabase/Electric/streams → writes still hit production. `arbe --local debug env` warns about this split. - Installed CLI disagrees with the checkout → `arbe --version` shows its baked commit. Run from source (`bun apps/cli/src/cli.ts …`) or rebuild with `arbe upgrade` while proving a CLI fix. - A form submission returns the `Cross-site … form submissions are forbidden` body → CSRF, not authentication. `apps/www/src/hooks.server.ts` owns the guard; bearer requests are its deliberate exemption. ## Verify a deployment Check each runtime changed by the commit: ```sh curl -fsS -H 'Cache-Control: no-cache' https://arbe.0sk.ar/api/version fly logs -a arbe-backstage --no-tail | tail ``` `arbe thread director` (bare) ends with a `backstage:` line naming every worker that heartbeated in the last three minutes, or a `⚠ backstage offline` warning. Read it before blaming a slow bot: a thread `behind` with `no lease held` and no beating backstage is a dead runner. Local `bun run dev` starts no worker: a healthy local HTTP preview cannot rescue a stopped production backstage. The backstage has no HTTP service, so flyctl has no health check and will call a machine that came up *stopped* "in a good state" and exit 0. CI therefore runs `verify-deploy` after every deploy (every machine must be `started`), and the deploy job is serialized with a concurrency group: two pushes minutes apart once ran two deploys on top of each other, the second replaced the machine and left it stopped, and CI was green for 90 minutes of silence (2026-09-03). `fly status -a arbe-backstage` shows the machine state; `fly machine start -a arbe-backstage` is the remedy. `/api/version` identifies www only. The backstage version appears in its startup logs. Dispatch turns from `packages/core/dispatch/` execute inside the backstage, so a core dispatch change ships with the backstage, not with www. A cross-package change may require both deployments. After deploy, post a fresh entry, then inspect `arbe thread trace ` and backstage logs. Static assets can remain edge-cached briefly; retry with `Cache-Control: no-cache` and `Pragma: no-cache` before concluding the new asset did not ship. ## Backstage image fails to build The `Deploy backstage` step fails with `Unknown lockfile version` while www and docs build green: the image's bun is older than whatever wrote `bun.lock`. Keep `FROM oven/bun:-slim` at or ahead of the bun that rewrites the lockfile (a local `bun install` on a newer bun upgrades it silently). www and docs going green hides the fallout — bots, workflows and every dispatch turn run in the backstage, so read the last green Build with `gh run list` before believing a dispatch change is live. The docs are served at `https://arbe.0sk.ar/docs` through www's `DOCS` service binding, so they need both workers. Deploy `docs` before `www` — `scripts/deploy.ts` already orders them that way — and verify with `curl -sI https://arbe.0sk.ar/docs/system/architecture/` plus `curl -s https://arbe.0sk.ar/llms.txt | head -1`. ## Docs site builds stale `astro build` caches per-package in `apps/docs/node_modules/.astro`, and that cache survives a `dist/` wipe. An edit to `astro.config.mjs` — a remark plugin, `base`, `site` — can therefore appear to do nothing while the build reports success. Symptom: some pages carry the new output and some carry the old. Clear both caches before concluding the config is wrong: ```sh rm -rf apps/docs/.astro apps/docs/node_modules/.astro apps/docs/dist bun run --filter '@arbe/www-docs' build ``` ## svelte-check reports a file that does not exist `svelte-check --tsgo --incremental` transpiles the Svelte tree to disk under `apps/www/.svelte-kit/.svelte-check/` and reuses it. Change `apps/www/tsconfig.json` — especially `files`/`include` — and the overlay tsconfig written next to that cache keeps the old shape, so the check fails with `Error: File '.../.svelte-check/svelte/.d.ts' not found` for a path nothing references any more. The source is correct; the cache is not. ```sh rm -rf apps/www/.svelte-kit bun run --filter '@arbe/www' check ``` ## `X is not defined` only in the browser `@arbe/core` is shared by the server, the CLI, and the browser, and nothing marks which modules are browser-safe. `Buffer`, `process`, and `node:*` imports type-check fine (globally typed) and pass `bun run test` (bun has them), so they only fail once a real browser loads the page. Before importing a core module from browser code, check it for Node-only APIs; use web-standard ones (`TextEncoder`, `crypto.subtle`, `Uint8Array`) or keep the code behind a `lib/server` or `.server.ts` boundary. ## A `redirect()` in a page load 500s on a cold load `apps/www/src/routes/+layout.ts` sets `ssr = false`, so every `+page.ts`/`+layout.ts` load runs in the browser. A `redirect()` thrown from a **page** load on the first render of a URL does not navigate: the `Redirect` reaches `handleError` instead, the analytics console logs `client exception: {status: 307, location: …}`, and the visitor gets the 500 page. Redirect from the component with `goto(href, { replaceState: true })` in an `$effect`, or from `hooks.server.ts`, not from a page load. While you are there: deleting a route file Vite is currently serving crashes `bun run dev` with an ENOENT for the deleted path. Write the replacement first, delete second, and expect to restart the dev server if you get the order wrong. ## CLI output truncates at 65536 bytes bun buffers stdio into its node-stream shim, and `process.exit` throws away whatever the 64KiB pipe buffer has not swallowed — so a piped CLI command stops at exactly 65536 bytes (arbe-7ef2, arbe-cbd4). Three things make this hard to see, all measured on bun 1.3.14: - **`console.log` is not exempt.** It writes straight to the fd in a bare script, but is buffered like everything else once anything has touched `process.stdout` — which the CLI does on every run. It also never calls the stream's `write` method, so a `process.stdout.write` shim never sees it. - **`Bun.spawn({ stdout: 'pipe' })` does not reproduce it.** It returns the full bytes even from a truncating binary; the first fix for arbe-cbd4 shipped green and broken on such a test. Reproduce and assert through a real shell pipe: `sh -c 'bun cli … | cat | wc -c'`. - **`writableLength` / `needDrain` read 0** right after a 70KB write that returned `false`, so any "is something pending?" gate never fires. The fix is `installSyncStdio()` (`apps/cli/src/output.ts`), called once at the CLI entry: it routes both the streams and the console methods through a blocking `write(2)`. Two traps if you touch it — bun leaves piped stdio non-blocking, so a full pipe raises `EAGAIN` (retry, don't let it throw), and blocking defers SIGINT until the reader drains. ## Website performance Slow page opening or thread switching → [measurement recipe](website-performance.md). Start with `bun run apps/www/scripts/profile-routes.ts --help`; the profiler is read-only. Keep the browser, fixtures, and source quiet during each run. Save JSON, compare with `apps/www/scripts/compare-performance.ts`, and repeat both versions before claiming a gain. Local Vite measurements do not establish cold-start or production performance. [Baseline, botlab fixtures, and evidence](../../../apps/www/proofs/performance/2026-09-07/README.md) — including a repeat that disproved an apparent speedup. ### Thread creation `POST /api/threads` answers with a `Server-Timing` header naming every remote wait it did — `scope` (membership gate), `parent` (what `createThread` waited on for the resolved parent; ≈`scope` because the gate is handed over in flight), `stream` (durable-stream create), then `insert` / `config` / `participants`. Read it from curl, the browser network panel, or the bench harness: ```sh cd apps/www bun run scripts/bench-thread-create.ts --house --bot bun run scripts/bench-thread-create.ts --house --bot --runs 7 --json bun run scripts/bench-thread-create.ts --origin https://arbe.0sk.ar --house --bot ``` Use a disposable house — every run creates a real thread, and they are deleted afterwards unless `--keep`. Three shapes are measured: `house` (house parent, pregenerated id, seeded participant), `dm` (agent parent with an always-reply config seed), and `onboarding` (bot, membership, config-seeded interview thread, and greeting). Onboarding bots are deleted after their threads, so repeated benchmarks do not consume the house spawn cap. What the numbers said in Aug 2026 (arbe-76f9): each Postgres round trip costs 45–75 ms, and **durable-stream create costs 450–1000 ms** — more than everything else combined, on localhost and in production alike (raw `PUT` to Electric measures the same, so it is the provider's cost, not ours). Anything that only reorders our own Postgres trips is therefore worth ~100-200 ms at most; the stream is the floor while thread creation waits on it (arbe-5660). ### Repair localhost TLS Run `portless doctor` first. `bun run dev:trust` retries Portless's local certificate-authority trust; it does not repair an unhealthy proxy or route. A foreign/shared `bun run dev` process is a blocker: ask its owner rather than stopping or restarting it. ## Dispatch The diagnostic pipeline is: ```text conversation entry → wake_thread on configured queue → claim thread lease → read past decided_offset → decision → selected turn or nobody → tagged outcome → advance offset → catch up again ``` The mechanism is documented in [dispatch](../chat/dispatch.md) and [thread director](../chat/thread-director.md). Debug from the durable evidence: ```sh arbe thread diagnose arbe thread trace arbe thread entries list --json # lease renewals, catch-up passes, decisions, and turn internals are backstage logs: fly logs -a arbe-backstage --no-tail | tail ``` Picks and silence over new conversation write `signal.thread_director.decision`, whether chosen by rules or a model; a bot attempt then writes a tagged `signal.thread_director.outcome`. Bookkeeping-only windows advance `decided_offset` without another decision. Current selected-turn skip reasons are: | Reason | Meaning | |---|---| | `filtered` | the selected turn produced nothing more specific | | `budget_exceeded` | the house spend gate refused the selected turn | | `empty_reply` | the model completed without usable final text | `arbe thread director ` distinguishes caught-up, behind, and live-lease states. `arbe thread trace ` shows persisted decisions, outcomes, dispatch signals, and historical signal types. Reader copy for selected-turn failures and skips lives in [`reader-copy.ts`](../../../packages/core/dispatch/reader-copy.ts). ### Presence cues Replay the durable signals that drive queued/thinking/working state: ```sh arbe thread entries list --json | \ (cd apps/www && bun run scripts/replay-presence.ts) ``` The pre-signal “Deciding who replies…” and its stall message are client state, owned by [`composer-cues.ts`](../../../apps/www/src/lib/composer-cues.ts). A stall reports `client.error` because no terminal or progress signal arrived after a durable post. ### Dispatch latency Start with `arbe thread trace --house `. Each completed bot reports total trigger-to-completion time separately from its own turn; the outer `runner` is bookkeeping, not another reply. An explicit trigger entry id wins over later messages, so a human posting during generation cannot make the reply appear faster. Older inferred anchors say `estimated`; missing referenced entries stay unknown. Handoff is part of pickup, not an additional delay to add. New director decisions record the effective human-burst pause and the measured decision duration. One participant bot defaults to no pause; explicit scope settings win, and multi-bot threads retain the app default. This pause is not measured sleeping time: worker pickup and reads can overlap that interval. New turns report every model call's first nonempty text/thinking/tool delta and first text delta, measured from that call's start; these are provider-stream observations, not the moment text appeared in the browser. No delta means `not observed`, while old turns say the timing was not recorded. Use the trace's PostHog link for the same turn's usage analytics. System-prompt and tool-schema sizes are characters, not token estimates; message/tool counts explain why a clean conversation still has a large input. PostHog receives these counts and timings, never the exact prompt, answer, or request body. Historical exact context cannot be reconstructed from analytics. Check existing telemetry before proposing another capture store. The busy bar follows durable prepare/generate/tools/save transitions, with an indeterminate active stage. The initial choose stage includes the human-burst wait; it does not claim the director has started a model call. Progress writes are included in Arbe processing, outside measured provider and tool time. `arbe thread trace ` pairs each persisted `signal.thread_director.decision` with dispatch start/completion and its tagged outcome. It reports pickup, handoff, reply-turn phases, tokens, and cost when those fields exist. Phase fields do not sum to the full duration; use fresh production samples rather than old medians when investigating latency. New `modelMs` measures provider-call wall time and `toolsMs` measures tool execution; historical `llmMs` includes both plus transcript writes. The completed-line Arbe remainder excludes both, but begins at dispatch start: pickup and handoff still contain earlier Arbe work and intentional debounce. Compare no-tool turns with the same model and record the debounce setting; do not treat a shorter configured pause as faster processing. Before judging a scenario’s silence or speaker selection, verify each bot’s effective thread mode. Creating a thread with one bot seeds an `always` override even if that bot is ambient at house scope. The scenario runner must set and read back its declared modes on every new thread before posting; scenario 19 otherwise answers “Perfect” correctly for its actual configuration while failing the ambient expectation. For a workflow instruction followed by `no eligible bots`, inspect the recorded author kind before changing prompts or adding mentions. Workflow instructions are authored by the house System agent and the assigned bot has an `always` override; resolving System as unknown prevents that bot from owing a reply. ### Prompt caching defaults Bot replies build their context in `packages/core/dispatch/dispatch.ts`, then `tool-loop.ts` calls pi-ai's `stream`/`streamSimple`. They request usage, pass the thread id as `sessionId`, request `cacheRetention: 'long'`, and send OpenRouter's explicit `x-session-id` header. The header pins a conversation even before its first cache hit and supplies Z.AI's upstream session affinity; `prompt_cache_key` alone is OpenRouter's fallback routing key. Neither option guarantees a hit or forces an unsupported provider to cache. Defaults by upstream provider ([OpenRouter reference](https://openrouter.ai/docs/guides/best-practices/prompt-caching), checked 2026-09-13): - **Z.AI (including GLM-5.3-Flash) and DeepSeek (including V4 Flash):** automatic, no `cache_control` needed. The existing read-only evidence in `arbe thread trace tplxswluvwvu --house xkwkqzyovpwm` includes GLM reads of 4,608 tokens and DeepSeek reads of 7,424, interspersed with misses. Zero writes does not mean caching is disabled. - **OpenAI, xAI, Moonshot:** automatic; OpenAI requires at least 1,024 prompt tokens. OpenAI GPT-5.6+ also offers explicit breakpoints, but Arbe leaves its automatic caching enabled. Groq's automatic caching is model-specific (Kimi K2). - **Anthropic:** requires opt-in. pi-ai already marks the system prompt, last tool, and conversation tail. Arbe's `long` requests a one-hour TTL where supported, rather than the default five minutes; writes cost more than ordinary input. Minimum cacheable length depends on the Claude model. - **Alibaba:** explicit five-minute markers on the supported unsuffixed model ids listed in `dispatch/reply.ts`; pi-ai marks the system, last tool and conversation tail. Snapshot endpoints are deliberately excluded. The policy applies to both bundled and dynamically resolved models, without changing routing. - **Gemini 2.5+:** implicit caching, with model-specific minimum sizes; Arbe does not request its optional explicit cache storage or pay its storage charge. Other models/endpoints have no blanket caching guarantee. The reusable prompt starts with identity, platform and authored instructions; changeable thread title/tags/status and per-turn directives follow them. No current clock or trace id is injected ahead of that prefix. Tool schemas have a stable order within a turn; changing enabled tools or reaching the forced final answer without tools changes the provider's prefix. Sliding the conversation window also invalidates some cached history. Retention, eviction and routing can still cause misses. Trace and usage ledger use pi-ai's reported `cached_tokens` and `cache_write_tokens`, not estimates; ordinary input excludes both. A reported zero is not proof that the provider cannot cache. Do not infer a cache hit from `cacheRetention` or prompt size. After deploying backstage, send two nearby replies with the same bot and tools in a fresh test-house thread, then inspect `arbe thread trace --house `: look for positive `cache read`, including later calls within a tool turn. Never post into the read-only evidence house above. The offline wire regression is `dispatch/prompt-caching.test.ts`; it verifies headers, Alibaba markers and usage mapping through the installed pi-ai adapter, not a simulated cache hit. ### A turn is stalled or missing its terminal `thread diagnose` returns the trigger entry and trace id. Correlate them with the durable task and its attempts: ```sh # run from packages/ bunx supabase db query --linked "select task_id, state, attempts, max_attempts, enqueue_at from absurd.t_dispatch where task_name='wake' and params->>'threadId'='' order by enqueue_at desc" -o table bunx supabase db query --linked "select run_id, attempt, state, claimed_by, claim_expires_at, failure_reason from absurd.r_dispatch where task_id='' order by attempt" -o table # The trace's cheap record lives in three real surfaces: `arbe thread trace ` # for per-turn LLM timings, `(cd apps/www && wrangler tail arbe)` for request-level # logs from the www Worker, and the `usage_events` ledger (joined by trace_id) for # spend. There is no `posthog-cli`; query the ledger directly: bunx supabase db query --linked "select seam, key_source, model_ref, input_tokens, output_tokens, cost_usd, created_at from usage_events where trace_id='' order by created_at" -o table ``` Interpret the failure shape before changing a timeout: - Turn cap → the core runner aborts the selected turn and writes `signal.dispatch.failed`; the director records a failure outcome. - Paid decision failure → the wake retries when retryable; when the budget is spent, or the model names a bot that isn't a candidate, the decision fails visibly (a failure outcome with the reason), never a silent nobody. - Lease loss → the stale pass stops before another append; another wake or the behind scan catches the thread up. - Lease contention → the second wake returns quietly; the pass holding the lease keeps reading until the thread is caught up, and the janitor is the backstop. - `[thread-director.scan] stranded thread ` → the five-minute janitor had to wake a thread, which means a wake was lost or a pass ended early. In a healthy system this line never appears; find out why that thread fell behind (`arbe thread trace ` shows the gap as a slow pickup marked `⚠`). - `[thread-director.pass]` with repeated `settlement:stay` → inspect the tagged outcomes for that decision; three transient failures or one permanent failure should let the offset advance. - Backstage process loss → the task claim and thread lease expire, then a fresh claim reconstructs settlement from the stream. Turn caps live in `packages/core/dispatch/reply-turn.ts`; lease intervals live at the top of `apps/backstage/src/thread-director-wake.ts`. Redelivery reuses persisted decisions and counts tagged outcomes from the stream. Spend is joined by the turn's trace id. `usage_events` is the ledger; PostHog is the second sink. See [observability](observability.md) and [analytics](analytics.md#usage--money). ## Env-bound dispatch Separate “did dispatch run?” from “could the sandbox answer?”: ```sh arbe env diagnose arbe thread diagnose arbe sandbox view # live daytona state arbe x -s -- bash -lc 'tail ~/arbe-pi-runner.log; tail ~/pi-run.log' ``` `run_command` results appear as `pi.tool_result` entries. For a complete proof, `bun run scripts/remote-dispatch.ts [] [--local]` plants a sandbox-only nonce and requires the bot to read it back. See [the manual runbook](../../../tests/remote-dispatch-prompt.md) and [Daytona](../sandboxes/sandbox-daytona.md). Detached work runs on a child thread. Start diagnosis there; snapshots from `thread entries list` are safer for agents than `--follow`, which intentionally never returns. A child with no terminal should be checked against the runner logs above. If a thread keeps reaching an old sandbox after rebinding environments, follow [stale thread sandbox pointers](../sandboxes/environments.md#stale-thread-sandbox-pointers). ## Backstage `arbe wf runs` heads its listing with the specific condition and its remedy — read it before reaching for a restart, because only two of the four cases are the backstage's fault: | Header line | What it means | Remedy | | --- | --- | --- | | runner offline | no heartbeat in three minutes | restart the backstage | | beating but has not claimed N runs | runnable work sitting unclaimed — wedged | restart the backstage | | parked with no await anchor | no `wf_run_threads` row, so nothing can emit the wake event | cancel and respawn; see arbe-4478 | | waiting on a person / a bot turn | parked on `ctx.awaitEvent`; a human gate waits as long as it takes, a bot turn past an hour is lost | open the run thread | | `` has retried N times | uncapped or pathological retry ladder | fix the task — a restart only makes it retry sooner | A high `retried N×` on a finished row is the same signal after the fact. ```sh fly logs -a arbe-backstage --no-tail bun run --filter '@arbe/backstage' deploy ``` Use the package deploy script. Bare `fly deploy` from `apps/backstage` has the wrong Docker build context. A post-deploy `Cannot find module './x.js'` crash means the partial image missed a transitive import; `bun run check` includes `check:backstage-image`, which identifies imports outside the Docker `COPY` set or missing production dependencies. Director model experiments are process-wide backstage variants. Set both values explicitly, deploy/restart through the package script, then post a fresh botlab entry and read the durable proof: ```sh fly secrets set -a arbe-backstage \ ARBE_DIRECTOR_MODEL=openai/gpt-5.6-terra \ ARBE_DIRECTOR_THINKING_LEVEL=high bun run --filter '@arbe/backstage' deploy fly logs -a arbe-backstage --no-tail | tail # directorModel=… directorThinking=… arbe thread trace # model=… thinking=… ``` `ARBE_DIRECTOR_MODEL` accepts either `openai/gpt-5.6-terra` or `openrouter/openai/gpt-5.6-terra` form and normalizes both to the canonical startup/trace ref. The default `off` sends no reasoning parameter, leaving reasoning behavior to the provider default. Unset both secrets to return to the code-owned defaults (`openai/gpt-5.6-terra`, `off`): `fly secrets unset -a arbe-backstage ARBE_DIRECTOR_MODEL ARBE_DIRECTOR_THINKING_LEVEL`, then deploy/restart and verify the same two surfaces. Blank values also act as unset. Startup fails before polling on malformed nonblank values, unknown models, or unsupported model/thinking combinations. Only paid decision witnesses show `model=… thinking=…`; model-free decisions say `no llm time` and omit the configured variant. Production backstage on Fly consumes all background work, including work submitted through localhost. `bun run dev` runs only www/API and docs; `apps/www/tmp/dev.log` contains local HTTP logs. Read worker activity with `fly logs -a arbe-backstage` and durable decisions/outcomes with `arbe thread trace --house `. Trace records the executing backstage SHA, including failures; unstamped older records say unknown. A later deployment does not change this attribution. Check local HTTP/auth readiness separately from production worker heartbeat. Before proving changed background code, deploy it from `main`, then submit fresh work in a test house. Test houses isolate records, not deployment effects. An invalid queued payload must fail before execution; the worker log identifies its kind/id, validation details, running SHA, and deploy-first hint. A schema-valid payload can still depend on changed semantics, so a passing shape check does not replace coordinated deployment. Every failed command, surprising flag, and source-only discovery is debugging-surface feedback. Fix the command, proof, skill, or canonical doc so the next investigation starts with the discriminator.