# Agent Benchmark Runner Implementation Plan

**Status:** done

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build a packaged benchmark CLI that runs deterministic mock or live OpenAI-compatible comparisons for both invocation modes and writes secret-safe JSON/Markdown reports.

**Architecture:** Add a benchmark application module for config, case catalog, run models, timing and aggregation; a reporting infrastructure module for redaction/rendering/atomic writes; and a thin presentation CLI. Each run uses `DebugRuntime.start_session()` with a temporary SQLite store and timing chat-client wrapper.

**Tech Stack:** Python 3.11+, asyncio, Pydantic, argparse, httpx-based existing OpenAI-compatible client, SQLite, pytest, uv.

---

### Task 1: Benchmark configuration and case catalog

**Files:**
- Create: `src/agent_lab/application/benchmark.py`
- Create: `tests/test_benchmark_config.py`
- Modify: `tests/test_tool_invocation_comparison.py`

- [x] Write failing tests for strict schema version, URL validation, known case/mode IDs, positive runs, and rejection of secret fields.
- [x] Verify RED with `uv run pytest tests/test_benchmark_config.py -q`.
- [x] Implement strict Pydantic models `BenchmarkConfig`, `BenchmarkTarget`, `BenchmarkCase`, `BenchmarkExpectation`, and immutable built-in case catalog.
- [x] Implement `build_benchmark_request(case, mode, model)` using the existing `DebugRunRequest`, identical params, and built-in enabled tools.
- [x] Expose deterministic mock rounds from the production case catalog and refactor the existing comparison test to reuse the catalog instead of duplicating prompts/expectations.
- [x] Run config and comparison tests, then commit with `feat: add benchmark case catalog`.

### Task 2: Timing wrapper and isolated benchmark runner

**Files:**
- Modify: `src/agent_lab/application/benchmark.py`
- Create: `tests/test_benchmark_runner.py`

- [x] Write failing tests for provider TTFT versus visible TTFT, per-call elapsed/usage, persisted wall time, detected events/sources, tool/fallback counts, semantic failures, partial-failure continuation, and temporary-database isolation.
- [x] Verify RED with `uv run pytest tests/test_benchmark_runner.py -q`.
- [x] Implement `TimingChatClient` around the existing stream protocol, recording first item, first message delta, usage, elapsed, and fallback classification.
- [x] Implement deterministic `MockBenchmarkChatClient` that never constructs HTTP transport and emits mode-correct streams for the six cases.
- [x] Implement `BenchmarkRunner.run()` over case → mode → iteration, creating/closing one runtime, chat client, and temporary SQLite store per run.
- [x] Implement semantic evaluation from output, audit, and usage ledgers and return ordered `BenchmarkRunResult` records without stopping after an individual failure.
- [x] Run runner/config/comparison tests, then commit with `feat: run isolated agent benchmarks`.

### Task 3: Aggregation, reporting, and CLI

**Files:**
- Create: `src/agent_lab/infrastructure/benchmark_reporting.py`
- Create: `src/agent_lab/presentation/benchmark_cli.py`
- Create: `tests/test_benchmark_reporting.py`
- Create: `tests/test_benchmark_cli.py`
- Modify: `pyproject.toml`

- [x] Write failing tests for linear-interpolated p50/p95, failed-sample exclusion, low-sample warnings, redaction, JSON/Markdown consistency, atomic writes, CLI overrides, mock/live API-key requirements, exit codes 0/1/2, and continued report writing after run failures.
- [x] Verify RED with `uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q`.
- [x] Implement aggregation into one typed `BenchmarkReport` model shared by JSON and Markdown.
- [x] Implement centralized redaction and 2 KiB error truncation before any rendered output.
- [x] Implement one atomically published report bundle containing matching `report.json` and `report.md` files.
- [x] Implement argparse with `--config`, `--base-url`, `--model`, repeatable `--case`/`--mode`, `--runs`, `--output-dir`, and `--mock`; apply `CLI > JSON` and environment-only API key.
- [x] Register `agent-lab-benchmark = agent_lab.presentation.benchmark_cli:main` and verify `--help`.
- [x] Run reporting/CLI/runner tests, then commit with `feat: add benchmark reports and cli`.

### Task 4: Examples, documentation, and final verification

**Files:**
- Create: `benchmarks/single-target.example.json`
- Modify: `.gitignore`
- Modify: `README.md`
- Modify: `docs/plans/todo-52-agent-benchmark-runner.md`
- Modify: `docs/plans/todos.md`

- [x] Add a secret-free example covering both modes and all six cases; ignore `outputs/benchmarks/`.
- [x] Document `.env`, config copy/edit, mock command, live command, output fields, exit codes, fairness controls, and the limits of scripted/mock adapters.
- [x] Run `agent-lab-benchmark --config benchmarks/single-target.example.json --mock --output-dir /tmp/agent-lab-benchmark` and inspect both reports.
- [x] Run focused benchmark tests, `uv run pytest`, CLI help, app import, `node --check`, and `git diff --check`.
- [x] Perform specification review then code-quality review; add Todo 52.x rows for post-execution fixes.
- [x] Record evidence and Workflow Evolution findings, mark Todo 52 done, and commit with `docs: close benchmark runner todo`.

## Final Evidence

Verified on 2026-07-14 (Asia/Shanghai):

- benchmark suite: `196 passed in 1.19s`;
- full suite: `601 passed in 3.67s`, with one existing Starlette/httpx deprecation warning;
- `agent-lab-benchmark --help`: exit `0`;
- application import prints `Agent Lab`;
- `node --check src/agent_lab/presentation/static/app.js`: exit `0`;
- `git diff --check`: exit `0`;
- zero-network mock CLI: `12 attempted / 12 passed / 0 failed`;
- final JSON: `/tmp/agent-lab-benchmark-final-20260714/20260713T194850Z/report.json`;
- final Markdown: `/tmp/agent-lab-benchmark-final-20260714/20260713T194850Z/report.md`;
- both parallel-mode runs report `parallel_events_overlapped=true`;
- both web-search runs report two model calls and one tool execution;
- no mock run has missing token usage.

Specification and code-quality reviews found no remaining Critical or Important
issues after Todos 52.1-52.13. The remaining non-blocking risks are documented
below.

## Residual Risks and Limits

- No live provider request was run because no real benchmark target or credential
  was supplied; live latency, usage support, semantic adherence, and provider
  tool-stream compatibility still require operator verification.
- The example uses one run per case, so its 60 p95 warnings are expected and the
  mock percentile values are not suitable for ranking architectures.
- Atomic bundle publication uses a relative directory symlink and is verified on
  macOS; Windows symlink behavior is not verified.
- An interruption inside `mkdtemp()` may leave an empty hidden bundle directory.
  A close failure during a narrow consumer-pause cancellation window may be
  logged as a secondary async-generator exception. Neither case changes the
  primary cancellation result or exposes a mixed public report.

## Workflow Evolution Update

The requirement loop exposed three durable workflow lessons:

- paired filesystem artifacts need one explicit atomic publication boundary;
  repeated two-file rollback fixes were replaced by a single report bundle;
- benchmark plans must define clock boundaries, missing-data semantics, and
  observable concurrency proof before implementation, not only metric names and
  event counts;
- parallel subagents need shared-contract integration checks when one task changes
  an output layout consumed by another task's tests.

These lessons are recorded in this task report and the benchmark design. No
`AGENTS.md` change was made because the repository's existing PAOU, todo-first,
TDD, per-todo commit, and post-review follow-up rules already caused the issues to
be found and corrected; adding more global workflow rules would duplicate the
current core process.