Observability
How the system witnesses its own operation: latencies, decisions, spend, failures. Not content (that’s streams), not structural change (that’s mutations) — see primitives for where signals sit in the formal layer.
Four layers, picked by the question you’re asking:
| Layer | Answers | Storage | Retention |
|---|---|---|---|
| Run state | what ran, what’s running, what happened last | Postgres + thread streams, via HTTP API | durable |
| Usage | who spent what, on whose key, at what cost | usage_events + PostHog | durable / plan limits |
| Lifecycle | aggregate trends — signups, reply rates, activity | signals on threads + PostHog mirror | durable / plan limits |
| Cloudflare logs | live debugging | CF dashboard, wrangler tail | ~72h (free tier) |
Run state is remote-only: the CLI has no local database; Postgres and the thread’s durable stream are the sole sources of truth.
Usage: recordUsage() writes paid work to the ledger and PostHog. A thread is an AI session; each turn or coding run is a trace containing model generations and tool spans. Details: analytics and LLM keys.
Lifecycle: typed signal.<entity>.<verb> entries on threads, normally mirrored to PostHog by track() when the agent opted in. Content-free operational counts can be always-on; house.created is, so creation-rate monitoring covers every API client. Call shape and vocabulary: analytics.
Cloudflare logs: console output streams to the CF dashboard and wrangler tail (observability.logs.enabled in wrangler config). Console statements use bracket prefixes: [entries], [submit-entries], [usage]. Dispatch’s own prefixes — [dispatch-wake], [dispatch-worker], [director.run], [reply-turn.in-process], [dispatch.turn] — come from the backstage, so read them with fly logs -a arbe-backstage, not wrangler tail.
Dispatch writes durable signal.dispatch.* entries and mirrors them to PostHog with the same trace id. Director decisions and outcomes also stay on the thread.
Paid director responses create a usage row and an AI generation with bounded timing, model, reasoning, and result fields. Calls with no reported spend remain visible only as a failure class. Delivery is best-effort; durable retries are separate work.
PostHog receives timing, model/tool usage, failures, cost, and app ids. It never receives prompts, answers, tool arguments/results, auth data, browser sessions, emails, or names. GeoIP is disabled.
See analytics, debugging, system/dispatch.