# Observability

How the system witnesses its own operation: latencies, decisions, spend, failures. Not content (that's streams), not structural change (that's mutations) — see [primitives](../../thinking/primitives.md) for where signals sit in the formal layer.

Four layers, picked by the question you're asking:

| Layer | Answers | Storage | Retention |
|---|---|---|---|
| Run state | what ran, what's running, what happened last | Postgres + thread streams, via HTTP API | durable |
| Usage | who spent what, on whose key, at what cost | `usage_events` + PostHog | durable / plan limits |
| Lifecycle | aggregate trends — signups, reply rates, activity | signals on threads + PostHog mirror | durable / plan limits |
| Cloudflare logs | live debugging | CF dashboard, `wrangler tail` | ~72h (free tier) |

Run state is remote-only: the CLI has no local database; Postgres and the thread's durable stream are the sole sources of truth.

Usage: `recordUsage()` writes paid work to the ledger and PostHog. A thread is an AI session; each turn or coding run is a trace containing model generations and tool spans. Details: [analytics](analytics.md#usage--money) and [LLM keys](../access/llm-keys.md).

Lifecycle: typed `signal.<entity>.<verb>` entries on threads, normally mirrored to PostHog by `track()` when the agent opted in. Content-free operational counts can be always-on; `house.created` is, so creation-rate monitoring covers every API client. Call shape and vocabulary: [analytics](analytics.md).

Cloudflare logs: console output streams to the CF dashboard and `wrangler tail` (`observability.logs.enabled` in wrangler config). Console statements use bracket prefixes: `[entries]`, `[submit-entries]`, `[usage]`. Dispatch's own prefixes — `[dispatch-wake]`, `[dispatch-worker]`, `[director.run]`, `[reply-turn.in-process]`, `[dispatch.turn]` — come from the backstage, so read them with `fly logs -a arbe-backstage`, not `wrangler tail`.

Dispatch writes durable `signal.dispatch.*` entries and mirrors them to PostHog with the same trace id. Director decisions and outcomes also stay on the thread.

Paid director responses create a usage row and an AI generation with bounded timing, model, reasoning, and result fields. Calls with no reported spend remain visible only as a failure class. Delivery is best-effort; durable retries are separate work.

PostHog receives timing, model/tool usage, failures, cost, and app ids. It never receives prompts, answers, tool arguments/results, auth data, browser sessions, emails, or names. GeoIP is disabled.

See [analytics](analytics.md), [debugging](debugging.md), [system/dispatch](../chat/dispatch.md).