Skip to content
View as .md

Testing

Prove it works first by using arbe; tests come second.

Proof is a thread id. arbe breaks where a unit test can’t follow: real models, the director choosing who speaks, durable streams, sandboxes, the deployed worker. So a change is done when a real thread in a test house, with real bots and real messages, shows it working: a director scenario (bun tests/run-scenario.ts <n> --house new), a recipe in packages/skills/verify-arbe/features/, or a tests/*.md prompt. Reports and commit bodies lead with the thread ids. When no scenario or recipe reaches the behavior, add one; that beats a test. packages/supabase/tests/verify-*.sql proves the DB contracts the same way, against the real DB.

Tests lock in what using arbe showed is worth protecting. A unit test earns its keep only when it covers logic that can break in a way a live run won’t catch fast, and that a rewrite-with-identical-behaviour would still pass:

  • Parsers / decoders / projectors over real captured fixtures. Real pi JSONL, real byte frames, real wire payloads (core/pi/events.test.ts, sandbox/src/daytona/client.test.ts). A synthesized “valid” input only proves the happy path you imagined.
  • Decision functions with real branching. Eligibility, reconciliation, selection, status transitions — pure inputs → asserted outputs, injected clock (decideStuckThreadReconciliation, director/decision.test.ts).
  • State machines and concurrency. Fencing, coalescing, retry ladders.
  • Crash and race paths you can’t stage live. A redelivered turn, a write that landed but was never acknowledged, two runs racing for one box. Name the crash in the test; if a thread could show it, it isn’t one of these.
  • Escaping / injection / serialization contracts where every character is load-bearing (errors/arbe-error.test.ts, shell-escaping, electric-shape.test.ts rejection rules).
  • Regression tests for a real past bug, with a one-line note on what broke. These are gold.

Never mock collaborators to fake a flow you could run. A test that stubs streams, the store, an Absurd context, a model, or our own modules to walk a normal path proves the stubs, not arbe. Run the flow in a thread instead.

A review flags a missing live proof first, before style or coverage.

Everything else is noise. Do not write — and delete on sight:

  • Wiring and orchestration tests. Anything that stubs our own modules, a fake database, or a fake stream to walk a path a thread would show. If it needs a mock of arbe itself, the test proves nothing a type error wouldn’t.
  • Mapper and shape tests. Row → domain type, field-presence checks, “the builder includes the fields the builder includes”. Schema.parse(validInput) tests Zod, not us.
  • Restated early returns, trivial builders, constant maps, call-logs.
  • Case-by-case enumeration of a union where one representative case plus the unknown-variant fallback covers the actual risk. Ten near-identical toMatchObject blocks over ten subtypes is shape-testing, not coverage.

Size is a smell, not a rule: a test file larger than its module is almost always testing the code’s shape. Aim well under 1x source LOC; get there by cutting cases that can’t fail independently, not by compressing formatting.

When a test survives

  • Assert the outcome, not the choreography. Payload at a real external boundary, yes; that a call happened, no.
  • Mock only what you can’t run: network, clock, incidental fs. Mocking our own modules means the seam is wrong — extract the pure decision or delete the test.
  • Name tests for the behaviour (cooldown + human trigger resets eligibility, not classify works).
  • Factor repeated fixture boilerplate into a small builder, keep each case’s asserted data inline, drop fixture fields no assertion reads.

Server seams

A route handler or daemon loop is wiring, proven by running it. When one of its crash or race paths needs a test, the ordered effects belong in a unit that takes its collaborators. submitThreadEntries(deps, input) in packages/core/dispatch/submit-entries.ts is the shape. Test that unit, never the route handler, and never by stacking module mocks around it. dispatch/submit-entries.test.ts is a current copy source for ordered, effectful IO such as thread-director-wake.ts, dispatch/reply-turn-runtime.ts, or anything that spends or writes durably:

  • One fixture builder per seam returns { config, clients, posted }. Dependencies a path must not touch are a Proxy that throws on any read, so “read the idempotency witness before spending” is proven by a loud failure instead of a call count.
  • Fire-and-forget sinks are faked at their real external boundary, not mocked in our own module: configureUsage() with a fake usage_events insert, then await flushUsage() before asserting rows. Reset it in afterEach — the config is module-level.
  • Prove a crash-between case by failing the write and asserting what survives, then re-running the same idempotency key to pin what a retry costs. Fail it both ways — nothing written, and written-but-unacknowledged — or the cheap half is all you have covered.
  • Assert the order, not just the survival. Push each effect into one shared log and assert the sequence. “A usage row exists” passes just as happily when the spend is recorded before the durable write it was ordered after, which is the bug the ordering exists to prevent.

Where verification actually lives

  1. A real thread first. Director scenarios, verify-arbe recipes, the tests/*.md prompts (tests/README.md) and the probes in scripts/. A change to orchestration is verified by running it, not by re-faking its dependencies.
  2. SQL contracts in SQL. begin … rollback proofs in packages/supabase/tests/verify-*.sql against the linked DB — never a JS fake of postgrest.
  3. Unit tests last, only for the bar above.

Mechanics

  • packages/sandbox/src/daytona/generated/bundles.ts is gitignored build output, written by the root postinstall (bun install) and by the backstage image build. If it is missing, core tests and checks fail to load: run bun run --filter '@arbe/sandbox' build, then repeat. Running repository build scripts to regenerate derived artifacts is standing-authorized; it is not a blocker requiring user approval.

  • bun run test (scope: bun run --filter '@arbe/core' test). Never bare bun test — it bypasses the package script. Paths passed after a filtered package script are relative to that package (dispatch/submit-entries.test.ts, not packages/core/dispatch/submit-entries.test.ts).

  • Two runners, deliberately split by runtime: packages/* + apps/www on vitest (vi.mock), apps/cli on bun:test (mock). Match the runner already in the package — the boundary is the runtime, not preference (evaluated in arbe-883f).

  • Co-locate foo.test.ts next to foo.ts; shared fixtures in a __fixtures__ dir beside the tests that use them.