# Analytics

Three surfaces, picked by what the event *is* — never send the same fact through two of them by hand:

- **Signals** — durable in-app lifecycle record (`signal.<entity>.<verb>` on a thread).
- **`track()`** — PostHog mirror of lifecycle events, normally opt-in per agent; content-free operational counts such as `house.created` are always-on.
- **`recordUsage()`** — money. Not opt-in — billing attribution. See [usage](#usage--money).

## Lifecycle — signals + track()

Signals are the durable record: schema-validated `signal.<entity>.<verb>` payloads written atomically with the mutation. `signal.thread.*`, `signal.thread_director.*`, `signal.entry.*` and `signal.dispatch.*` land on the thread itself; `signal.house.*`, `signal.files.*` and `signal.environment.*` on the scope's primary thread. `track()` is the PostHog mirror — called at the route boundary, fire-and-forget via `waitUntil`, only when the agent opted in (or always-on when the event is classified operational).

```
mutation ──┬─► postThreadEntry  signal.house.created   (durable, queryable, in-app)
           └─► track(...)       house.created          (PostHog, always-on operational count)
```

Adding a lifecycle event is two independent decisions: post a typed `signal.*` from the core mutation if it should appear on a thread; call `track()` at the route if it should appear in PostHog. Both, either, or neither. The one rule: anything worth recording durably belongs in the core schema as a typed `signal.*` — never through `track()` alone.

The signal vocabulary lives in `packages/core/schemas/stream-entries/thread.ts` — families `house`, `thread`, `thread_director`, `entry`, `files`, `environment`, `dispatch` (28 kinds at the time of writing; the file is the list). To add a kind: extend the schema, post from the mutation. Most signals have **no** PostHog mirror; only `signal.dispatch.*` is mirrored wholesale (below), and a handful of `track()` events exist without any signal (`agent.created`, `member.joined`, `thread.created`, `user.signed_in`, …). Closing that gap deliberately is task arbe-e9b7.

`track()` (`apps/www/src/lib/server/track.ts`) auto-injects `route`, `method`, `request_id`, `caller_kind`, `release_sha`, `ts`; caller properties win on collisions before the PostHog sanitiser runs. Lifecycle events are consent-gated per agent (`agents.telemetry_opt_in`, surfaced at `/account/telemetry`) unless explicitly classified as content-free operational telemetry. `house.created` is always-on so the operator can see the platform's true creation rate; it carries the house id but not its name. The tri-state is resolved in one place — `apps/www/src/lib/telemetry-consent.ts` — and during alpha an unanswered `null`, plus signed-out traffic that has no row at all, counts as **opted in**; only an explicit `false` opts out. Flip `TELEMETRY_DEFAULT_OPT_IN` there to go opt-in-only. Privacy line: no content or identity — `track()` drops content/name/path fields before capture; app ids (UUIDs) are sent raw; the shared transport disables PostHog GeoIP enrichment.

**Errors** — `trackError(...)` from www's server `handleError` emits always-on `server.error` operational events. The backstage emits the same event from worker, Postgres, wake, reconcile, event-loop, and fatal-process boundaries. Both send runtime/source, release, error kind or code, and a hash of the message; raw messages and payload content stay in local runtime logs. The opted-in browser client emits `client.error` for SvelteKit render/navigation failures, `window.error`, unhandled promise rejections, and explicitly detected client failures, with the same raw-message exclusion. That is the browser's **only** product event: `apps/www/src/lib/analytics.ts` initialises posthog-js with pageviews, pageleave, autocapture, exception capture and session replay all off, and nothing reads feature flags or surveys — so PostHog's web analytics, replay, and flag surfaces are empty by design, not by accident. The browser `distinct_id` is a hashed `user:<sha256>`, the server's is the raw agent uuid; the two never join.

Verify the browser path end to end: `agent-browser eval "setTimeout(() => { throw new Error('probe') }, 0); Promise.reject(new Error('probe2'))"` on any page emits `client.error` for `window.error` and `window.unhandledrejection`, signed in or out. For `server.error`, throw from any route (a temporary `throw` in a `+server.ts` under `bun run dev` is enough). Ingestion lags ~20-60s; match your own probe by `message_hash` (djb2 of the raw message), not by timestamp.


Current PostHog lifecycle events: `user.signed_in`, `house.created`, `house.deleted`, `agent.created`, `thread.created`, `member.joined`, `environment.created`, `invite.created`, `invite.claimed`, `account.deleted`, `agent.self_deleted`, `house.volume.written`, `house.volume.deleted`, `house.volume.changeset_written`. This list drifts — the authoritative one is a query away ([Querying](#querying)).

**Dispatch mirror** — `publishDispatchSignal` (`packages/core/dispatch/signals.ts`) mirrors the signals it is handed to always-on PostHog events named by stripping the `signal.` prefix: `dispatch.started`, `dispatch.completed`, `dispatch.skipped`, `dispatch.attempt_failed`, `dispatch.failed`, and the progress beat `dispatch.phase` (`phase`, `round`). `signal.dispatch.session_started` and `signal.dispatch.pi_waiting` are posted through other paths and are **not** mirrored. Events carry thread, agent, trace, duration, status, and reason; failures are marked as AI errors.

PostHog groups AI work like this:

```text
thread       → session
selected turn or Pi invocation → trace
model call   → generation
 tool call   → child span
```

A resumed Pi invocation starts a new trace in the same child-thread session. The director call, selected turn, dispatch signals, tools, and usage share one trace id.

Verify a signal landed: `arbe thread entries list <thread-id>`. Verify the PostHog side with `posthog-cli` ([Querying](#querying)).

```
PUBLIC_POSTHOG_KEY      empty disables PostHog
PUBLIC_POSTHOG_HOST     defaults from PostHog
RELEASE_SHA             stamped on track() events and backstage server.error only; falls back to 'dev'.
                        Usage, dispatch and client.error events carry no release identity (see arbe-b676).
```

## Usage — money

`recordUsage(event)` (`packages/core/usage.ts`) is the single call for paid usage. Every spending seam calls it after the spend. Fire-and-forget, never throws, two sinks per event sharing one `trace_id`. Delivery is **best-effort**: there is no outbox and no retry, so a sink or process failure drops that event silently — the guarantee is one logical emission per spend, not one durable row. Durable idempotent delivery is deferred to task arbe-e5b3.

1. **`usage_events`** (Postgres, append-only) — the ledger and future enforcement source. Columns: `house_id`, `agent_id`, `thread_id`, `capability` (`llm`/`sandbox`/`file_index`, open set), `seam` (open string; live values are whatever `ast-grep -p "seam: '$S'"` finds — currently `director`, `reply`, `read_thread`, `ask_thread`, `thread_title`, `cron_parse`, `sandbox_reply`, `run_command`, `delegate_task`, `sandbox_provision`, `sandbox_exec`, `file_search`, `volume_vision`, `gif_search`; retired values like `gate` and `file_upload` stay in PostHog forever), `key_source` (`worker` = arbe pays, `house`/`env` = the house pays — the resolver merges house secrets and environment bindings into one overlay, so an env-bound key currently reports `house` and nothing emits `env`), `model_ref`, `input_tokens`, `output_tokens`, `cost_usd` (pi-ai's `calculateCost` against its own model catalog rates — an estimate, not the provider's reported charge; llm meta says which via `costSource`), `meta`.
2. **PostHog** (`packages/core/usage-posthog.ts`) — `$ai_generation` for LLM spend (feeds the AI Observability dashboards; no prompt/output content), `arbe_usage` for the rest. House attached as group, joined to the ledger row by `$ai_trace_id`. The AI span also carries `$ai_latency` and a *parsed* projection of `meta`: metadata reaches PostHog only when the whole object satisfies `DirectorUsageMetaSchema` (`packages/core/schemas/director-usage.ts` — bounded enums, booleans, finite nonnegative integers, no unknown keys). A poisoned or unknown bag projects nothing at all rather than leaking by default, and `meta.failed` becomes `$ai_is_error`.

Each model attempt is one generation; a paid response also creates one ledger row. Tool calls are child spans with timing and error status. Strict schemas allow only counts, timing, stop reasons, and tool names—never prompts, answers, arguments, or results. See `ReplyUsageMetaSchema` and `schemas/sandbox-ai.ts`.

Wiring: www's `hooks.server.ts` and the backstage's `index.ts` each call `configureUsage({supabase, posthogKey, posthogHost})` once at boot — the ledger needs a service-role client (`usage_events` RLS rejects request-scoped ones) and core can't read `$env`. Scripts call it themselves; unconfigured, `recordUsage` warns and drops the row. The www hooks also hand `flushUsage()` to `waitUntil` after every request so workerd doesn't cancel in-flight writes; the backstage has no request hook, so every in-process turn and every failure narration drains its own in a `finally`.

Paid director attempts are metered at the response, before strict parsing can reject a malformed, ineligible, or empty choice. Each billable response records one usage event through `packages/core/thread-director/thread-director-decision-usage.ts`; a call with no response, tokens, or cost writes no row and surfaces a bounded failure class.

Key-source semantics: [llm-keys](../access/llm-keys.md). Smoke probe: `bunx varlock run -- bun run packages/core/scripts/test-llm-tracking.ts` — one real LLM call, poll-asserts the `usage_events` ledger row (sink 1) and prints its trace id; sink 2 (PostHog) is confirmed in the AI Observability dashboards or with `posthog-cli exp query run` against that trace id ([Querying](#querying)).

## Querying

`posthog-cli` is authenticated (`~/.posthog/credentials.json`: `https://eu.posthog.com`, project `133397`). Two surfaces:

- **`posthog-cli exp query run "<HogQL>"`** — the workhorse. HogQL is ClickHouse SQL over an `events` table. Prints **JSON-lines arrays, no column names** — pipe through `jq`/`python3` when you need keys, or just remember your own `select` order. `--debug` prints the raw envelope. `exp query check "<HogQL>"` resolves field names against the live schema without running the query; use it before anything long.
- **`posthog-cli api <cmd>`** — PostHog's MCP tool catalog as a shell: `search <regex>`, `info <tool>`, `call <tool> '<json>'`. Use it for things HogQL can't do: `insight-create` / `dashboard-create` to persist a query as a tile, `docs-search`, `query-llm-traces-list`. Destructive tools require `--confirm`; preview with `--dry-run`. `posthog-cli api --agent-help` is the full guide.

**Shell gotcha:** every PostHog-native property starts with `$`, which fish and bash expand inside double quotes. The SQL's own string literals need the single quotes, so the working shape is double-quoted SQL with each property escaped: `"select properties.\$ai_model from events where properties.seam='director'"`.

### What's in there

| family | events | key properties |
| --- | --- | --- |
| lifecycle (`track()`, normally opt-in) | `house.created` (always-on), `thread.created`, `user.signed_in`, … | `route`, `method`, `request_id`, `caller_kind`, `release_sha`, `ts` |
| errors | `server.error` (always-on), `client.error` (opt-in) | `runtime`, `source`/`mechanism`, `code`/`error_kind`, `message_hash`, `route`, `status`, `release_sha`, `fatal` |
| dispatch mirror (always-on) | `dispatch.started`/`.completed`/`.skipped`/`.attempt_failed`/`.failed`/`.phase` | `thread_id`, `agent_id`, `trace_id`, `entry_id`, `task_id`, `attempt`, `max_attempts`, `duration_ms`, `http_status`, `reason`, `phase`, `round` |
| usage (always-on) | `$ai_generation` (LLM), `$ai_span` (tools), `arbe_usage` (sandbox/files) | `seam`, `key_source`, `house_id`, `agent_id`, `thread_id`, `trace_id`, `$ai_session_id`, `$ai_trace_id`, `$ai_span_id`, `$ai_parent_id`, `$ai_model`, `$ai_tools_called`, `$ai_tool_call_count`, token/cost/latency fields, plus the bounded director projection (`decision_kind`, `validation_outcome`, `stop_class`, `output_mode`, `candidate_count`, `configured_thinking_level`, `reasoning_tokens`, `cost_source`) |

`$ai_generation_summary` and `$ai_trace_summary` are PostHog-derived rollups, not ours — don't sum cost from them and from `$ai_generation` in the same query.

**Join keys.** `trace_id` (mirrored as `$ai_trace_id`) is the turn: it ties the director call, the reply call, every tool seam, and the dispatch terminals together, and it is the same id as the `usage_events.trace_id` ledger row. `house_id` is also on `$group_0`. `distinct_id` is the raw agent uuid (falling back to thread or house id), not a person — there is no useful person model, and the `groups` table is empty because we set `$groups` on events but never call `groupIdentify`. Filter houses with `$group_0`, not by joining `groups`.

### Recipes

```sh
# Is data arriving at all, and what kinds?
posthog-cli exp query run "select event, count() as c, max(timestamp) as last
  from events where timestamp > now() - interval 7 day group by event order by c desc"

# Where the LLM money goes, by seam. (30d: reply $1.36, director $1.21, gate $0.64)
posthog-cli exp query run "select properties.seam as seam, count() as calls,
  round(sum(properties.\$ai_total_cost_usd), 4) as usd
  from events where event='\$ai_generation' and timestamp > now() - interval 30 day
  group by seam order by usd desc"

# Director cost per model, with latency.
posthog-cli exp query run "select properties.\$ai_model as model, count() as calls,
  round(sum(properties.\$ai_total_cost_usd), 4) as usd, round(avg(properties.\$ai_latency), 2) as lat
  from events where event='\$ai_generation' and properties.seam='director'
    and timestamp > now() - interval 7 day group by model order by usd desc"

# Are director decisions landing? (needs the meta projection — see caveats)
posthog-cli exp query run "select properties.decision_kind as kind,
  properties.validation_outcome as outcome, count() as n
  from events where event='\$ai_generation' and properties.seam='director'
    and timestamp > now() - interval 7 day group by kind, outcome order by n desc"

# Spend per house, or per turn.
posthog-cli exp query run "select \$group_0 as house, count() as calls,
  round(sum(properties.\$ai_total_cost_usd), 4) as usd
  from events where event='\$ai_generation' and timestamp > now() - interval 7 day
  group by house order by usd desc limit 20"

# One turn, end to end.
posthog-cli exp query run "select timestamp, event, properties.seam, properties.\$ai_model,
  properties.\$ai_total_cost_usd, properties.duration_ms
  from events where properties.trace_id='<trace-id>' order by timestamp"

# Why dispatches don't run. `reason` mixes enums (no_mode, gate_no, filtered,
# debounced, no_targets) with raw provider errors.
posthog-cli exp query run "select properties.reason as reason, count() as n
  from events where event in ('dispatch.failed','dispatch.skipped')
    and timestamp > now() - interval 30 day group by reason order by n desc limit 20"
# Errors by surface and stable message group.
posthog-cli exp query run "select event, properties.runtime, properties.source,
  properties.code, properties.message_hash, count() as n
  from events where event in ('server.error','client.error')
    and timestamp > now() - interval 7 day
  group by event, properties.runtime, properties.source, properties.code,
    properties.message_hash order by n desc limit 50"
```

A `trace_id` from any of these opens directly in the AI Observability UI, which renders the turn as a span tree:
`https://eu.posthog.com/project/133397/ai-observability/traces/<trace-id>`.

### Caveats

- **Money lives in the ledger.** `usage_events` (Postgres) is the billing source of truth; PostHog is the exploration surface. Both are best-effort (no outbox — see above), so a PostHog total is a lower bound, and the two can disagree. Reconcile through `trace_id`.
- **Token sums undercount.** `$ai_cache_reporting_exclusive` is true: `$ai_input_tokens` excludes cached reads, which land in `$ai_cache_read_input_tokens` / `$ai_cache_creation_input_tokens`. Aggregate `$ai_total_cost_usd`, not tokens. Some providers report no `$ai_latency` at all.
- **`seam` is an open string**, and PostHog keeps retired values forever — `gate` still holds ~$0.64 of history but stopped being emitted on 2026-08-15. Retired event names linger the same way: `house.file.uploaded` appears in the event list but nothing emits it since 2026-08-26. Always window your queries by time.
- **`release_sha` is not on everything, and its shape varies.** Only `track()` events and backstage `server.error` carry it, as either a full 40-char sha, a short 7-char one, `dev`, or null depending on where the process got it. Don't group by it across event families.
- **`dispatch.completed` fires twice per turn**, from `dispatch.ts` (per-bot: has `agent_id`, `trace_id`, `phases`) and from `reply-turn.ts` (outer backstage task: neither). Count turns with `uniq(properties.entry_id)`, not `count()`.
- **The director meta projection is new** — it started landing 2026-08-24, so `decision_kind` / `validation_outcome` are null on everything before that. A poisoned or unknown meta bag also projects nothing rather than partially (see [Usage](#usage--money)), so null means "not projected", never "no decision".
- **Lifecycle events are consent-gated** (alpha default: on — see above); usage and dispatch events are not. Never read a `track()` event count as a population count.
- `$ip` is still attached to every server event even though `$geoip_disable` suppresses enrichment.

## Code

One server-side PostHog transport: `capturePosthogEvent` in `@arbe/core/posthog` (direct POST to `/capture/` — workerd-safe; don't add `posthog-node`). Both `apps/www/src/lib/server/posthog.ts` (route capture) and `packages/core/usage-posthog.ts` (usage sink) delegate to it.

See [debugging](debugging.md), [observability](observability.md), [llm-keys](../access/llm-keys.md).

_Gaps: `signed_up` vs `signed_in` not distinguished (`ensureHumanAgent` is idempotent); `signal.agent.created` and `signal.member.joined` aren't in core's vocabulary yet — currently PostHog-only; read-path activity not instrumented; the CLI emits nothing (CLI sign-in never fires `user.signed_in`); coding boxes, `delegate_task` and `run_command` record money and traces but no countable lifecycle; the backstage reports only failures, never a healthy heartbeat. The full list and the pick-or-reject call: task arbe-e9b7._

_Rejected: per-entry `thread.entry.created` PostHog events — too high-volume, weak dashboard value, messy consent; if activity trends are ever needed in PostHog, emit periodic aggregates instead._