Skip to content
View as .md

LLM models and keys

Users pick models, never providers. A model ref is provider/model-id (pi format, e.g. openrouter/anthropic/claude-haiku-4.5); the provider prefix only decides which key to look up. OpenRouter is the default: one key covers every model namespace. Direct-provider refs (anthropic/<id>, google/<id>) work only if a key for that provider resolves.

Model resolution

For every LLM turn (bot reply, director decision, sandbox pi turn):

thread config `model` → agent `model` → `DEFAULT_BOT_MODEL` (cheap OpenRouter preset)

Thread = “try this model for this conversation”, set rarely; it pins every participant. Agent = durable preference. OpenRouter ids resolve from pi-ai’s bundled catalog first, then OpenRouter’s live catalog, so a new model needs no arbe release; an unknown id still fails loudly. The director’s decision pins its own model id (DIRECTOR_DECISION_MODEL_ID in packages/core/dispatch/reply.ts) and takes only the provider from the thread.

Code: resolveReply in packages/core/dispatch/reply.ts; catalog in packages/core/pi/openrouter-models.ts.

Key resolution

One precedence for replies, director decisions, titles, cron parsing and delegated coding agents:

house BYOK secret → funded house key → clear refusal

Both live in the same house secret, OPENROUTER_API_KEY: funded_by = 'arbe' means arbe minted it, null means the house brought its own. Nothing on this path ever reads an operator model key. Without either, the refusal is visible in the thread: No model key for this house. Add OPENROUTER_API_KEY under house secrets. (HTTP: agent.not_configured, 400). Turn overlays are cached ~60s, so a rotated key lands within a minute; a failed secret read fails closed and is never cached.

Code: resolveTurnSecrets in packages/core/dispatch/turn-secrets.ts, the one resolver behind every seam.

Funded keys

Creating a house mints it an OpenRouter key under arbe’s account with a $5 lifetime ceiling and stores it as the house’s OPENROUTER_API_KEY. Minting is best effort and never blocks creation; the create response says funded_key: minted | failed | skipped. Setting your own key replaces funding and disables the minted key. Deleting the secret or the house deletes the key.

A few things here look wrong at first glance but are on purpose:

  • The funded key goes into the sandbox. A bot with run_command can read it with env. That is accepted: the key is capped at $5, scoped to one house, and revocable by rotation. The alternative (proxying every sandbox model call through arbe) was rejected as not worth the moving parts for an alpha.
  • The ceiling is not reserved credit. All houses share arbe’s balance, which can run out first. A house at its own ceiling sees budget_exceeded; arbe’s balance running out sees provider_balance_exhausted. Both name BYOK as the fix; BYOK 402s stay plain provider errors. OpenRouter reports both as 402 with no distinguishing body, so arbe asks /api/v1/auth/key for the calling key’s remaining allowance to tell them apart.
  • Rotation is not a refill. arbe house fund rotate --house <id> disables the old key, reads its usage, and mints a replacement with only the remaining allowance.
  • OpenRouter is the real limit; our ledger is just the estimate. The ledger’s cap (houses.spend_cap_usd, assertWorkerSpendAllowed) is a fast pre-check and an optimistic estimate: it undercounts when a worker dies before recording, and OpenRouter lets one request overshoot the ceiling. arbe house fund status --house <id> shows the key’s real usage next to the ledger, with the gap labelled “unexplained”; that gap is a reconciliation discrepancy, not proof of outside use.
  • Owners cannot raise their own cap. Operators run bun run --filter '@arbe/www' fund-limit -- <houseId> <usd>, which updates both the key ceiling and the ledger cap. fund-backfill [--dry-run] [houseId...] mints keys for houses that predate funding.

OPENROUTER_MANAGEMENT_KEY lives only on the www worker, with an optional OPENROUTER_WORKSPACE_ID naming the OpenRouter workspace keys are minted in (default workspace otherwise). It is never a house secret, sandbox variable or backstage secret; self-hosters who leave it unset get no funded keys and must BYOK. Code: packages/core/openrouter-keys.ts, apps/www/src/routes/api/houses/.

Spend attribution

Every paid seam calls recordUsage() (packages/core/usage.ts) with a key_source:

key_sourceWho paysCounts toward spend_cap_usd?
workerarbe (funded house keys or infrastructure keys)yes
househouse (BYOK secret)no
envhouse (env-bound secret)no

The resolver calls a minted key funded; recordUsage folds that into worker at that one boundary. Sandbox usage looks up the house key’s provenance (cached 60s, failures not cached). Cap checks run only before an arbe-funded spend: replies, director decisions, sandbox, cron parsing, GIF search, paid vision indexing. A BYOK house over cap keeps getting LLM turns; only worker-only seams refuse.

One gotcha worth knowing: budget_check is SECURITY DEFINER on purpose, because usage_events has no select policy and an invoker-rights gate would sum zero rows (migration 20260611150000).

Exceptions

  • Developer model proofs use ARBE_PROOF_OPENROUTER_API_KEY, passed explicitly.
  • File readings of images and scanned PDFs use the backstage’s GEMINI_API_KEY; they count against the house cap and are refused per file when over it. Text indexing makes no model calls (volumes).