docs: document agent benchmark runner

Problem:
- The benchmark runner had no checked-in single-target example or complete operator guide for mock/live execution, metrics, reports, and comparison controls.
- The design still described the obsolete flat two-file report layout.

Risk:
- Live benchmarks call the configured provider and may incur charges; the documentation now makes that boundary explicit.

.gitignore

@@ -70,3 +70,4 @@ target/
 *.db
 *.sqlite
 *.sqlite3
+outputs/benchmarks/

README.md

@@ -100,6 +100,114 @@ tools/events, configure ChatAgent and applicable EventAgent settings, set event
 loop/concurrency/batch limits, create or restore sessions, replay messages and
 audit data, and inspect per-call, per-turn, and per-session usage.
 
+## Benchmark CLI
+
+`agent-lab-benchmark` runs the same built-in scenarios through `dual_agent` and
+`chat_agent_tools`, then writes matching machine-readable and review-friendly
+reports. It runs in process against the production runtime and event kernel; it
+does not start FastAPI or add WebSocket/network timing to the measured turn.
+
+The API key is read only from `.env` or the environment through
+`AGENT_LAB_OPENAI_API_KEY`. Do not put a key in the benchmark JSON or commit it.
+The JSON config supplies the OpenAI-compatible `base_url`, `model`, run count,
+modes, and cases:
+
+```bash
+cp benchmarks/single-target.example.json benchmarks/single-target.local.json
+```
+
+Edit the copied file before a live run. The checked-in example deliberately uses
+`https://provider.example/v1` and `replace-with-model-id` placeholders and
+contains no secret.
+
+Start with the deterministic mock benchmark. This command does not require an
+API key and makes no network request:
+
+```bash
+uv run agent-lab-benchmark \
+  --config benchmarks/single-target.example.json \
+  --mock
+```
+
+For a live OpenAI-compatible provider, set the key locally, configure a real
+`base_url` and `model` in the copied JSON, and omit `--mock`:
+
+```bash
+export AGENT_LAB_OPENAI_API_KEY='replace-locally'
+uv run agent-lab-benchmark \
+  --config benchmarks/single-target.local.json
+```
+
+CLI values override JSON values. `--base-url`, `--model`, and `--runs` replace
+their scalar settings; repeat `--case` or `--mode` to replace the corresponding
+JSON list. For example, this still performs a zero-network mock run:
+
+```bash
+uv run agent-lab-benchmark \
+  --config benchmarks/single-target.example.json \
+  --mock \
+  --runs 3 \
+  --case web_search_two_answers \
+  --mode dual_agent \
+  --mode chat_agent_tools
+```
+
+The six built-in cases are:
+
+| Case | Expected behavior |
+| --- | --- |
+| `ordinary_chat` | One visible answer and no event. |
+| `device_volume_silent` | One volume event with silent success and no model follow-up. |
+| `calendar_schedule_template` | One schedule event followed by the deterministic confirmation template. |
+| `web_search_two_answers` | A fast first answer, one search event, and exactly one grounded second answer. |
+| `session_terminate` | One farewell, one terminate event, and a terminal session. |
+| `parallel_volume_schedule` | Volume and schedule events execute in one accepted batch with stable results. |
+
+Each run records provider TTFT (first provider stream item), visible TTFT (first
+non-whitespace user-visible delta), turn wall time, per-call model latency, tool
+handler latency, prompt/completion/total/cached tokens, model/fallback/tool call
+counts, and event names, sources, statuses, and semantic correctness. Provider
+TTFT and visible TTFT are intentionally separate; model and tool spans may
+overlap and are not added to manufacture a total latency.
+
+By default, reports are published as one complete bundle:
+
+```text
+outputs/benchmarks/<timestamp>/report.json
+outputs/benchmarks/<timestamp>/report.md
+```
+
+The timestamp path is a single relative symlink published only after both files
+are fully written and synced; collisions receive a numeric suffix. JSON is the
+machine-readable source of truth and Markdown is rendered from the same report
+model.
+
+Exit codes are stable:
+
+- `0`: all runs passed and reports were written;
+- `1`: at least one request or semantic check failed, but reports were written;
+- `2`: CLI, config, environment, runner-cardinality, or report/output failure;
+- `130`: interruption.
+
+For a fair mode comparison, keep the provider/model, generation parameters,
+input cases, run count, enabled events, limits, timeouts, adapters, and run
+conditions unchanged. Use fresh paired runs and inspect failed samples instead
+of comparing only averages. p95 is reported with a low-confidence warning when
+fewer than 20 successful samples exist in a case/mode group.
+
+Interpret results within these limits:
+
+- mock mode and the repository's scripted tests validate deterministic flow,
+  event calls, metrics, and report generation, not live provider behavior;
+- the runner is in process, so it does not measure FastAPI, WebSocket, service
+  transport, reconnect, or deployment overhead;
+- default event adapters are in-memory implementations and do not control a
+  device, create a real calendar entry, terminate a service conversation, or
+  search the public web;
+- the CLI does not calculate pricing or automatically declare/rank a winner;
+  evaluate latency, tokens, correctness, and operational tradeoffs together;
+- live runs send prompts to the configured provider and may incur API charges.
+
 ## WebSocket
 
 Connect to `ws://127.0.0.1:8000/ws/debug`. The smallest valid initial payload is:

benchmarks/single-target.example.json

@@ -0,0 +1,18 @@
+{
+  "schema_version": 1,
+  "base_url": "https://provider.example/v1",
+  "model": "replace-with-model-id",
+  "runs_per_case": 1,
+  "modes": [
+    "dual_agent",
+    "chat_agent_tools"
+  ],
+  "cases": [
+    "ordinary_chat",
+    "device_volume_silent",
+    "calendar_schedule_template",
+    "web_search_two_answers",
+    "session_terminate",
+    "parallel_volume_schedule"
+  ]
+}

docs/plans/2026-07-13-agent-benchmark-design.md

@@ -93,10 +93,15 @@ For each case/mode group, successful non-null samples produce p50 and p95 using
 
 ## Reports and Exit Codes
 
-One command writes matching timestamped files:
+One command writes a matching timestamped report bundle:
 
-- `outputs/benchmarks/<timestamp>.json` as the machine-readable source of truth;
-- `outputs/benchmarks/<timestamp>.md` rendered from the same report model.
+- `outputs/benchmarks/<timestamp>/report.json` as the machine-readable source of truth;
+- `outputs/benchmarks/<timestamp>/report.md` rendered from the same report model.
+
+Both files are fully written and synced in one private bundle directory before a
+single relative symlink publishes the timestamp path. Publication therefore has
+one atomic, no-overwrite boundary; a timestamp collision selects a numeric suffix
+instead of replacing an existing bundle.
 
 Exit codes: