docs: design agent benchmark runner

Problem: The invocation experiment lacked a configurable automated path for live or mock latency, token, event, and semantic evaluation.

Risk: Scripted mock runs validate orchestration only; live provider results remain sensitive to model randomness, cache state, and provider tool-stream behavior.

docs/plans/2026-07-13-agent-benchmark-design.md

@@ -0,0 +1,124 @@
+# Agent Invocation Benchmark Design
+
+**Status:** approved for implementation by the user's "feasible means proceed" instruction
+
+**Date:** 2026-07-13
+
+## Goal
+
+Provide a repeatable benchmark command where the operator configures an OpenAI-compatible `base_url`, `model`, and API key, then automatically runs the same business scenarios in `dual_agent` and `chat_agent_tools` modes and produces reviewable evaluation reports.
+
+## Chosen Architecture
+
+The first version is an in-process packaged CLI. It constructs the existing `OpenAICompatibleChatClient`, `DebugRuntime`, generic event kernel, built-in mock adapters, and SQLite session store directly. It does not start FastAPI or connect through WebSocket.
+
+This preserves the complete ChatAgent/EventAgent/event-kernel behavior while reducing benchmark orchestration noise. A later black-box benchmark may exercise WebSocket/network overhead after the in-process baseline is stable.
+
+Alternatives considered:
+
+- WebSocket black-box benchmark: validates the complete service boundary but adds server lifecycle, port, reconnect, and transport timing noise.
+- Provider-only benchmark: easiest latency measurement but cannot validate event detection, fallback, execution, result policies, or two-answer behavior.
+
+## Configuration
+
+The API key is read only from `.env`/environment through `AGENT_LAB_OPENAI_API_KEY`. Benchmark JSON must not contain secrets.
+
+```json
+{
+  "schema_version": 1,
+  "base_url": "https://provider.example/v1",
+  "model": "model-name",
+  "runs_per_case": 1,
+  "modes": ["dual_agent", "chat_agent_tools"],
+  "cases": [
+    "ordinary_chat",
+    "device_volume_silent",
+    "calendar_schedule_template",
+    "web_search_two_answers",
+    "session_terminate",
+    "parallel_volume_schedule"
+  ]
+}
+```
+
+CLI overrides JSON using `CLI > JSON`. Repeatable `--case` and `--mode` options replace the corresponding JSON lists. Live mode requires explicit base URL, model, and API key. `--mock` requires no API key and makes no network request.
+
+The first version intentionally supports one target per command. Multi-provider orchestration, parallel load, custom case DSL, pricing, CSV, and automatic winner ranking are out of scope.
+
+## Execution and Isolation
+
+Each case/mode/iteration run receives:
+
+- a unique session ID;
+- a dedicated temporary SQLite database;
+- the production `DebugRuntime` and default built-in in-memory adapters;
+- a timing wrapper around the configured chat client;
+- identical model parameters, enabled event definitions, event limits, and inputs across modes.
+
+The runner uses `start_session()` and consumes output through the first `turn_completed` or `error`, then reads persisted audit and usage data. It always closes the runtime/client and removes the temporary database.
+
+One failed run does not stop later combinations. Results preserve deterministic case → mode → iteration ordering.
+
+## Built-in Cases and Semantic Checks
+
+The benchmark catalog owns the same six production-shaped cases as the closeout matrix:
+
+- ordinary chat: one visible answer, no event;
+- device volume: fast first acknowledgement, one successful `device.volume.adjust`, no model follow-up;
+- schedule: fast acknowledgement, one successful `calendar.schedule.create`, one deterministic template;
+- web search: first answer, successful `knowledge.web.search`, exactly one second answer;
+- terminate: one farewell, successful `session.terminate`, terminal session;
+- parallel volume plus schedule: both events accepted, stable result order, one schedule template.
+
+Mock mode emits deterministic streams for these cases. Live mode relies on the configured model and marks semantic mismatches as failed runs rather than silently accepting them.
+
+Checks include answer count, first-answer ordering before tool completion, exact expected event names/source, normalized event status, model/tool/fallback counts, and terminal behavior.
+
+## Metrics
+
+Each run records:
+
+- initial provider TTFT from the first yielded provider stream item;
+- first visible delta TTFT measured from benchmark turn start;
+- pure model call elapsed values from the timing wrapper;
+- `turn_wall_time_ms` from the persisted turn ledger;
+- prompt, completion, total, and cached tokens;
+- ChatAgent model call count, EventAgent fallback count, logical tool count;
+- detected event names/sources, tool statuses, and handler latency;
+- semantic failures and a redacted/truncated error.
+
+Provider TTFT, visible TTFT, pure model elapsed, tool latency, and turn wall time remain separate. Overlapping spans are never summed to manufacture a total.
+
+For each case/mode group, successful non-null samples produce p50 and p95 using linear interpolation with `index = (n - 1) * q`. Markdown flags p95 as low confidence when fewer than 20 successful samples exist.
+
+## Reports and Exit Codes
+
+One command writes matching timestamped files:
+
+- `outputs/benchmarks/<timestamp>.json` as the machine-readable source of truth;
+- `outputs/benchmarks/<timestamp>.md` rendered from the same report model.
+
+Exit codes:
+
+- `0`: every run passed and both reports were written;
+- `1`: at least one request or semantic check failed, but reports were written;
+- `2`: CLI/config/environment/report-write failure;
+- `130`: interruption.
+
+## Security
+
+- API keys cannot appear in JSON configuration or report models.
+- Base URLs containing userinfo, query, or fragment are rejected.
+- Errors are redacted and truncated before stdout, JSON, or Markdown rendering.
+- Redaction covers configured key values, Bearer/API-Key forms, and common key/token parameters.
+- Reports do not include headers, raw SSE chunks, full provider bodies, or secrets.
+- The default benchmark adapters do not reach real device, calendar, session, or web-search services.
+
+## Testing
+
+- Config/CLI override and validation tests.
+- Deterministic mock end-to-end run with zero network construction.
+- Metric source separation and semantic validation tests.
+- Partial-failure continuation and exit-code tests.
+- Percentile, aggregation, JSON/Markdown consistency, atomic write, and secret-redaction tests.
+- Full repository regression and one generated mock report smoke.

docs/plans/todo-52-agent-benchmark-runner.md

@@ -0,0 +1,73 @@
+# Agent Benchmark Runner Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build a packaged benchmark CLI that runs deterministic mock or live OpenAI-compatible comparisons for both invocation modes and writes secret-safe JSON/Markdown reports.
+
+**Architecture:** Add a benchmark application module for config, case catalog, run models, timing and aggregation; a reporting infrastructure module for redaction/rendering/atomic writes; and a thin presentation CLI. Each run uses `DebugRuntime.start_session()` with a temporary SQLite store and timing chat-client wrapper.
+
+**Tech Stack:** Python 3.11+, asyncio, Pydantic, argparse, httpx-based existing OpenAI-compatible client, SQLite, pytest, uv.
+
+---
+
+### Task 1: Benchmark configuration and case catalog
+
+**Files:**
+- Create: `src/agent_lab/application/benchmark.py`
+- Create: `tests/test_benchmark_config.py`
+- Modify: `tests/test_tool_invocation_comparison.py`
+
+- [ ] Write failing tests for strict schema version, URL validation, known case/mode IDs, positive runs, and rejection of secret fields.
+- [ ] Verify RED with `uv run pytest tests/test_benchmark_config.py -q`.
+- [ ] Implement strict Pydantic models `BenchmarkConfig`, `BenchmarkTarget`, `BenchmarkCase`, `BenchmarkExpectation`, and immutable built-in case catalog.
+- [ ] Implement `build_benchmark_request(case, mode, model)` using the existing `DebugRunRequest`, identical params, and built-in enabled tools.
+- [ ] Expose deterministic mock rounds from the production case catalog and refactor the existing comparison test to reuse the catalog instead of duplicating prompts/expectations.
+- [ ] Run config and comparison tests, then commit with `feat: add benchmark case catalog`.
+
+### Task 2: Timing wrapper and isolated benchmark runner
+
+**Files:**
+- Modify: `src/agent_lab/application/benchmark.py`
+- Create: `tests/test_benchmark_runner.py`
+
+- [ ] Write failing tests for provider TTFT versus visible TTFT, per-call elapsed/usage, persisted wall time, detected events/sources, tool/fallback counts, semantic failures, partial-failure continuation, and temporary-database isolation.
+- [ ] Verify RED with `uv run pytest tests/test_benchmark_runner.py -q`.
+- [ ] Implement `TimingChatClient` around the existing stream protocol, recording first item, first message delta, usage, elapsed, and fallback classification.
+- [ ] Implement deterministic `MockBenchmarkChatClient` that never constructs HTTP transport and emits mode-correct streams for the six cases.
+- [ ] Implement `BenchmarkRunner.run()` over case → mode → iteration, creating/closing one runtime, chat client, and temporary SQLite store per run.
+- [ ] Implement semantic evaluation from output, audit, and usage ledgers and return ordered `BenchmarkRunResult` records without stopping after an individual failure.
+- [ ] Run runner/config/comparison tests, then commit with `feat: run isolated agent benchmarks`.
+
+### Task 3: Aggregation, reporting, and CLI
+
+**Files:**
+- Create: `src/agent_lab/infrastructure/benchmark_reporting.py`
+- Create: `src/agent_lab/presentation/benchmark_cli.py`
+- Create: `tests/test_benchmark_reporting.py`
+- Create: `tests/test_benchmark_cli.py`
+- Modify: `pyproject.toml`
+
+- [ ] Write failing tests for linear-interpolated p50/p95, failed-sample exclusion, low-sample warnings, redaction, JSON/Markdown consistency, atomic writes, CLI overrides, mock/live API-key requirements, exit codes 0/1/2, and continued report writing after run failures.
+- [ ] Verify RED with `uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q`.
+- [ ] Implement aggregation into one typed `BenchmarkReport` model shared by JSON and Markdown.
+- [ ] Implement centralized redaction and 2 KiB error truncation before any rendered output.
+- [ ] Implement atomic matching `.json`/`.md` writes under the configured output directory.
+- [ ] Implement argparse with `--config`, `--base-url`, `--model`, repeatable `--case`/`--mode`, `--runs`, `--output-dir`, and `--mock`; apply `CLI > JSON` and environment-only API key.
+- [ ] Register `agent-lab-benchmark = agent_lab.presentation.benchmark_cli:main` and verify `--help`.
+- [ ] Run reporting/CLI/runner tests, then commit with `feat: add benchmark reports and cli`.
+
+### Task 4: Examples, documentation, and final verification
+
+**Files:**
+- Create: `benchmarks/single-target.example.json`
+- Modify: `.gitignore`
+- Modify: `README.md`
+- Modify: `docs/plans/todo-52-agent-benchmark-runner.md`
+- Modify: `docs/plans/todos.md`
+
+- [ ] Add a secret-free example covering both modes and all six cases; ignore `outputs/benchmarks/`.
+- [ ] Document `.env`, config copy/edit, mock command, live command, output fields, exit codes, fairness controls, and the limits of scripted/mock adapters.
+- [ ] Run `agent-lab-benchmark --config benchmarks/single-target.example.json --mock --output-dir /tmp/agent-lab-benchmark` and inspect both reports.
+- [ ] Run focused benchmark tests, `uv run pytest`, CLI help, app import, `node --check`, and `git diff --check`.
+- [ ] Perform specification review then code-quality review; add Todo 52.x rows for post-execution fixes.
+- [ ] Record evidence and Workflow Evolution findings, mark Todo 52 done, and commit with `docs: close benchmark runner todo`.

docs/plans/todos.md

@@ -100,3 +100,4 @@
 | 50.1 | done | `docs/plans/todo-50.1-metric-key-migration-dedup.md` | Deduplicate legacy metric keys before creating the idempotency index so existing experiment databases remain openable. | Migration keeps the earliest row for each duplicate non-null session/metric key, creates the unique index, and remains safe under concurrent initialization. |
 | 51 | done | `docs/plans/todo-51-comparison-matrix-closeout.md` | Add repeatable comparison fixtures, update project documentation, and close the experimental implementation. | Both modes pass the same 12-run scenario matrix; focused (`231 passed`) and full (`417 passed`) tests, JS syntax, app import, and documentation checks pass. |
 | 51.1 | done | `docs/plans/todo-51.1-first-reply-and-protocol-matrix.md` | Require fast visible first replies for event scenarios and prove each mode uses its intended capability protocol surface. | Matrix asserts first reply ordering, no unintended second reply, dual compact event catalog/no provider tools, direct enabled schemas/no generated event catalog, and correct event source. |
+| 52 | in_progress | `docs/plans/todo-52-agent-benchmark-runner.md` | Add a configurable in-process benchmark CLI with deterministic mock cases and live OpenAI-compatible execution. | Mock/live runs compare both modes, measure provider/visible TTFT, wall time, tokens and event correctness, write secret-safe JSON/Markdown reports, and use stable exit codes. |