Volumes

A house has one canonical, versioned file tree. Files received through document intake arrive under documents/; files written by people, agents, and integrations can live anywhere in the tree. Text, PDFs, and images are indexed and searchable, and citations name the path and generation that produced the match.

write bytes to the volume
  → immutable blob + new volume generation
  → per-path reading state
  → bounded lexical chunks
  → current path/generation links
  → `search_files` / `arbe files search`

“Volume” is internal vocabulary. Humans and agents see files and folders — files is the page for them; this one is the storage and index internals.

Canonical bytes and history

Each house owns one volume. volume_generations records the complete path-to-blob tree at every generation; immutable bytes live under house-files/{houseId}/volumes/blobs/{git-blob-hash}. Overwriting a path creates a generation and keeps its earlier versions readable. Deleting a path is different: once deletion commits, that path and all of its versions must be unreachable and unrestorable through every product surface. Recreating the same path starts fresh history.

Writes compare-and-swap on the volume generation. A losing writer rebases prepared changes onto the new head, so disjoint writes both land. One changeset is one generation and cannot half-land.

Stored files remain byte-for-byte exact. Browser responses use nosniff; active formats such as HTML and SVG download rather than execute. CLI reads can pipe binary bytes without decoding them.

Deletion contract

Delete means gone from arbe, immediately after the deletion commits. Current and historical lists, byte and reading endpoints, file history, restore, CLI/SDK version reads, agent tools, and search must not recover the path or its earlier versions. Recreating the path creates unrelated new history. File-change announcements may retain the path, author, and time as content-free audit history, but never the deleted bytes or extracted text.

Derived readings and search data are removed. Content-addressed blob bytes are removed once no undeleted file or version refers to them; deletion must never break another file that shares identical bytes. Provider backups and bounded asynchronous cleanup may expire later, but deleted content is never product-readable while they do.

Deletion redacts the path from every earlier manifest in the same transaction, fences stale indexing work, and queues unreferenced blob bytes for bounded physical cleanup.

Code: packages/core/volume.ts, volume-blobs.ts, volume-supabase.ts, and schemas/volume.ts.

Search index

The index is a rebuildable projection of canonical generations. Each path exposes its current legibility:

• text — strict UTF-8 bytes are directly readable and indexable.
• reading — a durable reading exists and is indexable.
• extracting — extraction is still in progress.
• unsupported — the file type cannot be read by the index.
• failed — extraction was attempted but failed.

Strict UTF-8 decoding and text extraction from text-bearing PDFs happen locally. Raster images and scanned PDFs use an external vision model. Readings are stored by blob hash and extractor version, then reused wherever those bytes appear.

Readable content is split into bounded chunks for lexical indexing. Search is Postgres full-text search over chunk content plus path/filename matching; there are no embeddings and no external calls in the search path. Head links connect each chunk to its current path and generation, so deleted or replaced paths stop citing stale content. Failed vision extraction is isolated to that file rather than stalling later generations.

Ordinary keyword queries admit partial matches. Complete matches rank first, then unique term coverage and term rarity measured across distinct current paths in that house. Repeated words and extra chunks do not make a term more distinctive. Queries with explicit phrases, exclusions, or OR retain their websearch semantics. The search function owns this behavior in 20260905130000_volume_search_ranked_keywords.sql; indexing and bot prompts do not need to change.

The backstage processes volume-index tasks. packages/core/volume-extractor.ts owns PDF/image readings; volume-index.ts owns reading persistence, chunks, and head links. Tables and the search RPC live in 20260826100000_volume_file_index.sql and its follow-up migrations.

Only one file flow can reach Gemini:

image/scanned-PDF written or uploaded (web, CLI, API, bot)
  → generation commit → deferred DB trigger → volume-index task
  → vision reading once per new blob hash

Outside the backstage’s vision readings, the server makes no other Gemini calls: file search is lexical, ordinary reply turns resolve via OpenRouter, and blob cleanup makes no Google calls. Sandboxed bots calling explicit google/-prefixed models reach Gemini with their own house-supplied key.

GEMINI_API_KEY is the optional backstage key used for visual readings of images and scanned PDFs; without it, text indexing and search still work while those visual readings settle as failed. www never touches it, and it is never sent to a browser, model, or house.

Surfaces

CLI:

arbe files ls [folder]
arbe files cat <path> [--at <generation>]
arbe files put <path> [local-file|-]
arbe files search <query>
arbe files rm <path> --base-version <version>

HTTP:

Method	Path	Purpose
GET	/api/houses/:id/files	List the tree at a version
POST	/api/houses/:id/files	Apply a JSON member batch (changes: [{op: 'delete', path, baseVersion}] for deletes) or capability multipart batch
GET	/api/houses/:id/files/<path>	Read exact bytes
PUT	/api/houses/:id/files/<path>	Write one file
POST	/api/houses/:id/files/<path>	Restore one version
DELETE	/api/houses/:id/files/<path>?baseVersion=N	Delete one path at the version you saw; a newer version is a 409 and nothing is removed
GET	/api/houses/:id/file-history/<path>	Read path version history
GET	/api/houses/:id/file-reading/<path>	Read the settled extracted reading
POST	/api/houses/:id/file-search	Search the files and return citations
POST	/api/houses/:id/file-arrival	Intake one document under documents/

The TypeScript client exposes listHouseFiles, readHouseFile, readHouseFileReading, getHouseFileHistory, writeHouseFile, applyHouseFileChanges, arriveHouseFile, restoreHouseFileVersion, deleteHouseFile, and searchHouseFiles.

Agent tools split by capability:

• list_files, read_file, write_file, and delete_file operate on the versioned tree (volume-tools.ts).
• search_files searches the tree and returns path + generation citations (file-search-tools.ts).

A sandboxed agent may call write_file directly, and a delegated coding agent may publish a file itself through the capability-scoped curl door. Finishing a delegated run publishes nothing on its own; see sandbox files stay in the sandbox.

Operational proof

The production proof is the real surface: add a donor with arbe files put, observe its reading state, search a fact with arbe files search, and require a citation containing the path and generation. Exercise both a text PDF and an image.

A unique-token match proves indexing, not retrieval quality. bun run scripts/volume-search-ranking-proof.ts executes the migration’s SELECT body read-only on PostgreSQL against fixtures covering partial matches, distractors, ranking, query constraints, house scope, and current versions. Before applying the migration it validates the candidate query without creating a function. After applying it, repeat a fresh-bot memory question without supplying a filename and inspect its search results in the thread trace.

See files for the same tree described to the person using it, then storage for where the bytes sit, dispatch for the agent tools that write them, and LLM keys for the vision credential.