Skip to content
View as .md

Testing

The default is no unit test. arbe’s safety net is dogfooding plus proofs: the tests/*.md prompts drive the real deployed stack (which no unit test can), packages/supabase/tests/verify-*.sql proves the DB contracts, and apps/*/proofs/ scripts probe live behavior. Make the change work, verify it against the real system, and you’re done. Read this before adding or keeping a test.

The bar: a unit test earns its keep only when it covers logic that can genuinely break in a way dogfooding won’t catch fast, and that a rewrite-with-identical-behaviour would still pass:

  • Parsers / decoders / projectors over real captured fixtures. Real pi JSONL, real byte frames, real wire payloads (core/pi/events.test.ts, sandbox/src/daytona/client.test.ts). A synthesized “valid” input only proves the happy path you imagined.
  • Decision functions with real branching. Eligibility, reconciliation, selection, status transitions — pure inputs → asserted outputs, injected clock (decideStuckThreadReconciliation, director/decision.test.ts).
  • State machines and concurrency. Fencing, coalescing, retry ladders.
  • Escaping / injection / serialization contracts where every character is load-bearing (errors/arbe-error.test.ts, shell-escaping, electric-shape.test.ts rejection rules).
  • Regression tests for a real past bug, with a one-line note on what broke. These are gold.

Everything else is noise. Do not write — and delete on sight:

  • Wiring and orchestration tests. Anything that stubs our own modules to prove args pass through. If it needs a mock of arbe itself, the test proves nothing a type error wouldn’t.
  • Mapper and shape tests. Row → domain type, field-presence checks, “the builder includes the fields the builder includes”. Schema.parse(validInput) tests Zod, not us.
  • Restated early returns, trivial builders, constant maps, call-logs.
  • Case-by-case enumeration of a union where one representative case plus the unknown-variant fallback covers the actual risk. Ten near-identical toMatchObject blocks over ten subtypes is shape-testing, not coverage.

Size is a smell, not a rule: a test file larger than its module is almost always testing the code’s shape. Aim well under 1x source LOC; get there by cutting cases that can’t fail independently, not by compressing formatting.

When a test survives

  • Assert the outcome, not the choreography. Payload at a real external boundary, yes; that a call happened, no.
  • Mock only what you can’t run: network, clock, incidental fs. Mocking our own modules means the seam is wrong — extract the pure decision or delete the test.
  • Name tests for the behaviour (cooldown + human trigger resets eligibility, not classify works).
  • Factor repeated fixture boilerplate into a small builder, keep each case’s asserted data inline, drop fixture fields no assertion reads.

Server seams

A route handler or daemon loop is wiring. The ordered effects it wires up belong in a unit that takes its collaborators. submitThreadEntries(deps, input) in packages/core/dispatch/submit-entries.ts is the shape. Test that unit, never the route handler, and never by stacking module mocks around it. dispatch/submit-entries.test.ts is a current copy source for ordered, effectful IO such as thread-director-wake.ts, dispatch/reply-turn-runtime.ts, or anything that spends or writes durably:

  • One fixture builder per seam returns { config, clients, posted }. Dependencies a path must not touch are a Proxy that throws on any read, so “read the idempotency witness before spending” is proven by a loud failure instead of a call count.
  • Fire-and-forget sinks are faked at their real external boundary, not mocked in our own module: configureUsage() with a fake usage_events insert, then await flushUsage() before asserting rows. Reset it in afterEach — the config is module-level.
  • Prove a crash-between case by failing the write and asserting what survives, then re-running the same idempotency key to pin what a retry costs. Fail it both ways — nothing written, and written-but-unacknowledged — or the cheap half is all you have covered.
  • Assert the order, not just the survival. Push each effect into one shared log and assert the sequence. “A usage row exists” passes just as happily when the spend is recorded before the durable write it was ordered after, which is the bug the ordering exists to prevent.

Where verification actually lives

  1. Proofs and dogfooding first. tests/README.md explains the markdown proof prompts; apps/*/proofs/ hold executable probes; use arbe itself. A change to thin orchestration is verified by running it, not by re-faking its dependencies.
  2. SQL contracts in SQL. begin … rollback proofs in packages/supabase/tests/verify-*.sql against the linked DB — never a JS fake of postgrest.
  3. Unit tests last, only for the bar above.

Mechanics

  • packages/sandbox/src/daytona/generated/bundles.ts is committed, so a fresh clone has it; when it falls behind runner.ts / thread-mirror.ts, bun run --filter '@arbe/sandbox' check:drift says so and bun run --filter '@arbe/sandbox' build rewrites it. Running repository build scripts to regenerate derived artifacts is standing-authorized; it is not a blocker requiring user approval.

  • bun run test (scope: bun run --filter '@arbe/core' test). Never bare bun test — it bypasses the package script. Paths passed after a filtered package script are relative to that package (dispatch/submit-entries.test.ts, not packages/core/dispatch/submit-entries.test.ts).

  • Two runners, deliberately split by runtime: packages/* + apps/www on vitest (vi.mock), apps/cli on bun:test (mock). Match the runner already in the package — the boundary is the runtime, not preference (evaluated in arbe-883f).

  • Co-locate foo.test.ts next to foo.ts; shared fixtures in a __fixtures__ dir beside the tests that use them.