Skip to content
View as .md

Bot conversation storm proof — S15 full-cast (2026-08-30)

Scope: this was a focused S15-only run, not a comparable full-suite run. The suite in tests/director-evaluation-prompt.md gained a new scenario, S15 (full-cast storm: one fresh thread, all seven pinned cast bots, tools disabled in thread config, a bounded concurrent burst of five unmentioned canned messages, settle to skipped flurry_rest with no manual containment). Only S15 ran today; no other scenario ran, so nothing here is comparable to a full-suite table. Suite count updated 17 → 18 in the prompt and the README index. Paid live runs approved by Oskar in advance.

Three runs happened. Run 1 failed its gate (invalid). Run 2 was behaviorally green but is superseded: it ran without preventive tool isolation. Run 3 — fresh thread, tools disabled before any dispatch, explicit failure containment — is the verdict.

Run 1 — klrpowxwmosn: invalid (gate not met + unexpected file deletion)

Ran against the original S15 wording (“what’s one thing this project should stop doing?”). Routing and termination were correct (deciding beat -1→4, nova, beat_completed; follow-on beats bix and ada; flurry_rest at 1.4 m; no manual containment), but the run is not a valid PASS:

  • Burst gate not met. The suite requires one captured exit-0 status per background post. The poster loop and the wait loop ran in different shell invocations, jobs -p was empty in the second, and all five pid exit statuses were lost. Entry ids (5/5) and trigger lines (5/5) held; the exit-status leg did not. Treated as a gate miss, not worked around.
  • Unexpected house-file deletion. The actionable wording pulled bots into house files: ada ran read_file / delete_file tool rounds and deleted documents/thread-house-signals-mnlltolzxlnm-2026-08-26T15-36-11-929Z.md in botlab. CLI evidence verifies that it and the retained documents/thread-house-signals-mnlltolzxlnm.md were identical 176-byte files, and signal.files.changed records the deletion at generation 8. The deleted file was redundant, but the write was still an unintended scenario side effect.

Nothing from run 1 counts toward any verdict. Both prompt defects were fixed before rerunning.

Run 2 — wtvmnlnnszzt: behaviorally green, superseded

Fixed single-shell pid capture (5/5 exit 0), safe canned wording (polling-vs-webhooks tradeoff), zero tool calls, deciding beat -1→4 ada, follow-on eve and cleo, flurry_rest at 47.7 s with no containment. Behaviorally green — but it lacked preventive tool isolation: tools were merely hoped away by wording, and run 1 had already proven wording alone does not prevent tool use. Not the valid proof; superseded by run 3.

Run 3 — thread ywrymznvrsms (the verdict)

Fresh thread ywrymznvrsms (arbe-thread-ywrymznvrsms), name dir-eval abf6 S15 full-cast-storm r3, run tag abf6.

Preflight (third run): cast check PASS — cast matches the fixture. (first line endpoint https://arbe.0sk.ar · as oskarrough human a13364ba); untruncated arbe wf runs production summary ✓ workflow runner healthy — no runs stuck waiting (checked 26s ago) — the current healthy form the preflight now accepts; no warning lines, so the four pre-authorized stale conductor lines (workflow-conductor-oskar, workflow-conductor-781599ebd63278, workflow-conductor-r, workflow-conductor-cursor) never appeared and the one-day override went unused.

Tool isolation (before any dispatch). arbe config set thread ywrymznvrsms --patch '{"dispatch":{"tools":[]}}' — raw proof: patch: {"dispatch":{"tools":[]}}; resolved proof (config get --json): config.dispatch.tools: [] resolved over the house/app chain. No other config write was made.

Canned inputs (all five concurrent)

storm 1: one line — a tradeoff between polling and webhooks.
storm 2: one line — a tradeoff between polling and webhooks.
storm 3: one line — a tradeoff between polling and webhooks.
storm 4: one line — a tradeoff between polling and webhooks.
storm 5: one line — a tradeoff between polling and webhooks.

Gate — all three counts met before any trace analysis

  • Exit statuses 5/5: one single-shell invocation spawned all five posts into a pid array and waited each captured pid in that same invocation: pids 1406202–1406206, all exit 0.
  • Entry ids 5/5, one complete {id, threadId} JSON per post: storm 1 c6c7add6-363f-4c15-95c8-8709d7f97dde · storm 2 2460d5a6-3507-41f0-b39a-59ae25b3d835 · storm 3 23aeb41b-1cb5-43c9-9f94-2834ee617c3e · storm 4 56707e9e-fb88-44b6-a204-10df8f83ff3d · storm 5 af054f47-f3d1-4c84-81da-4cf83420db74
  • Trigger lines 5/5 in arbe thread trace ywrymznvrsms (arrival order 2, 3, 4, 5, 1 within 423 ms).

Participant snapshot

At creation and after the run — identical, delta empty, no joins, no removals: nova 555539bb · ada 87269073 · eve ec861612 · bix ecf8b358 · dot e483bd47 · cleo a464541f · probe2-chess 741fbea7 · oskarrough (human, trigger author).

Trace ranges and closers

4 beat_started, each with exactly one closer, offsets contiguous, no gap, covering all five posted entries (-1→4 spans all five entry ids):

beatrangedecisioncloserspeaker
1 (deciding)-1→4@ada 1.5s llm/1.7s total (paid)beat_completed (-1→4)ada — full coalescing, one speaker
2 (follow-on)4→11@eve 1.2s llm/1.2s totalbeat_completed (4→11)eve (context)
3 (follow-on)11→18@cleo 1.0s llm/1.1s totalbeat_completed (11→18)cleo (context)
418→25nobody no_candidates 1ms total, no llm timedirector nobody (18→25) + skipped flurry_rest—

0 coalesced (the offsets are the coalescing proof: one deciding beat spanning all five entries), 0 beat_failed, 0 fallback_*, 0 signal.dispatch.failed.

Waits, retries, containment

  • One wait cycle for the burst: timeout 120 arbe thread entries read exit 0, then five 25 s entries list --follow settle probes (each exit 124 by design, output to file, never piped through tail).
  • skipped flurry_rest confirmed in the trace after probe 5 (at 1.1 m); 30 s later the trace was unchanged and thread diagnose --json shows stage: completed, verdict: ok.
  • Containment was not needed: no sixth probe, no participant removal, no manual intervention. The failure-containment rule (remove every cast bot, verify no candidates remain, record FAIL, stop) stayed unexercised.
  • Retries: none.

Metrics (thread totals, every reply on the thread)

  • Replies: 3 (ada, eve, cleo). Input 2,842, output 1,859.
  • Reasoning tokens: 1,725 (45 ada + 832 eve + 848 cleo), from entries list --json → each pi.assistant entry’s payload.message.usage.reasoning — the surface exists and is now the canonical report source (the old n/a row was wrong).
  • Cost: trace reply lines $0.0001 + $0.0003 + $0.0003 = $0.0007 (usage payload exact: $0.00069). Director-decision spend: n/a — not exposed at this layer.
  • Context growth visible per reply in: ada 1,122 (all five storm entries, tools disabled), eve 1,157, cleo 563.
  • Paid decision aggregate uses total decision time, not llm time: totals 1.7 s, 1.2 s, 1.1 s (n=3, n<5 → raw sorted: 1.1, 1.2, 1.7). Run 2’s report had wrongly used the llm split.
  • Post→reply for the deciding beat: 7.2 s. Whole turns: ada 4.2 s, eve 22.8 s, cleo 29.5 s.
  • Trace id: n/a — in-process director threads expose no traceId via thread diagnose --json.

Tool calls and house-state writes

Zero from this thread. entries list --json contains no toolCall content in pi.assistant entries and no pi.tool_result entries; all three turns report toolRounds: 0; dispatch.tools=[] was resolved before dispatch. Scope note: this proves no tool or file-change entry originated from this thread — it is thread-scoped evidence and says nothing about house-wide changes made by other actors. No participant removal, no config change beyond the one permitted write.

Verdict (run 3)

PASS. Hard invariants: no INELIGIBLE_SPEAKER, no MISSING_TERMINAL (4/4 beat_started closed: three beat_completed, one nobody + skipped flurry_rest), no DUPLICATE_SPEAKER (one speaker in the deciding beat), no MALFORMED_DECISION, no LOST_OBLIGATION, no FLURRY_BROKEN (three consecutive bot turns — ada, eve, cleo — then rest). Judgement: acceptable, any of the seven permitted. Termination: PASS — stopped on its own; flurry_rest at 1.1 m after three consecutive bot turns, trace unchanged 30 s later, diagnose verdict ok, participants unchanged.

Anomalies and DX findings

  1. Run 1 (klrpowxwmosn) invalid: burst gate not met (exit statuses lost across shells) and the original wording drove ada’s tool rounds and deletion of a CLI-verified duplicate 176-byte botlab house file. Kept as evidence of both failure shapes.
  2. Run 2 (wtvmnlnnszzt) behaviorally green, superseded: no preventive tool isolation — wording is not a control. dispatch.tools=[] is now the S15 precondition.
  3. Preflight now accepts the current healthy production summary (✓ workflow runner healthy — no runs stuck waiting (checked <n>s ago)); the four pre-authorized stale conductor lines never appeared in any run today; the override went unused.
  4. The corrected single-shell pid-capture pattern worked first try in both runs 2 and 3.

Files

  • tests/director-evaluation-prompt.md — S15 (tools-off config proof before dispatch, safe wording, thread-scoped no-tool pass condition, failure containment that bounds spend, suite count 18); shared S12 burst snippet (single-shell pid array, per-post exit statuses); Do-not exception for the one S15 config write; preflight healthy-summary wording; reasoning-tokens report source corrected to payload.message.usage.reasoning.
  • tests/README.md — index line: eighteen scenarios, storm subcase named.
  • docs/thinking/bot-conversation-storm-proof-2026-08-30.md — this report.

Threads, in run order: klrpowxwmosn (invalid: gate miss + file deletion) · wtvmnlnnszzt (behaviorally green, superseded — no tool isolation) · ywrymznvrsms (valid verdict).