agent-lab/docs/plans/todo-52-agent-benchmark-runner.md

Agent Benchmark Runner Implementation Plan

Status: done

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a packaged benchmark CLI that runs deterministic mock or live OpenAI-compatible comparisons for both invocation modes and writes secret-safe JSON/Markdown reports.

Architecture: Add a benchmark application module for config, case catalog, run models, timing and aggregation; a reporting infrastructure module for redaction/rendering/atomic writes; and a thin presentation CLI. Each run uses DebugRuntime.start_session() with a temporary SQLite store and timing chat-client wrapper.

Tech Stack: Python 3.11+, asyncio, Pydantic, argparse, httpx-based existing OpenAI-compatible client, SQLite, pytest, uv.

---

Task 1: Benchmark configuration and case catalog

Files:

• Create: src/agent_lab/application/benchmark.py
• Create: tests/test_benchmark_config.py

• Modify: tests/test_tool_invocation_comparison.py

• [x] Write failing tests for strict schema version, URL validation, known case/mode IDs, positive runs, and rejection of secret fields.

• [x] Verify RED with uv run pytest tests/test_benchmark_config.py -q.

• [x] Implement strict Pydantic models BenchmarkConfig, BenchmarkTarget, BenchmarkCase, BenchmarkExpectation, and immutable built-in case catalog.

• [x] Implement build_benchmark_request(case, mode, model) using the existing DebugRunRequest, identical params, and built-in enabled tools.

• [x] Expose deterministic mock rounds from the production case catalog and refactor the existing comparison test to reuse the catalog instead of duplicating prompts/expectations.

• [x] Run config and comparison tests, then commit with feat: add benchmark case catalog.

Task 2: Timing wrapper and isolated benchmark runner

Files:

• Modify: src/agent_lab/application/benchmark.py

• Create: tests/test_benchmark_runner.py

• [x] Write failing tests for provider TTFT versus visible TTFT, per-call elapsed/usage, persisted wall time, detected events/sources, tool/fallback counts, semantic failures, partial-failure continuation, and temporary-database isolation.

• [x] Verify RED with uv run pytest tests/test_benchmark_runner.py -q.

• [x] Implement TimingChatClient around the existing stream protocol, recording first item, first message delta, usage, elapsed, and fallback classification.

• [x] Implement deterministic MockBenchmarkChatClient that never constructs HTTP transport and emits mode-correct streams for the six cases.

• [x] Implement BenchmarkRunner.run() over case → mode → iteration, creating/closing one runtime, chat client, and temporary SQLite store per run.

• [x] Implement semantic evaluation from output, audit, and usage ledgers and return ordered BenchmarkRunResult records without stopping after an individual failure.

• [x] Run runner/config/comparison tests, then commit with feat: run isolated agent benchmarks.

Task 3: Aggregation, reporting, and CLI

Files:

• Create: src/agent_lab/infrastructure/benchmark_reporting.py
• Create: src/agent_lab/presentation/benchmark_cli.py
• Create: tests/test_benchmark_reporting.py
• Create: tests/test_benchmark_cli.py

• Modify: pyproject.toml

• [x] Write failing tests for linear-interpolated p50/p95, failed-sample exclusion, low-sample warnings, redaction, JSON/Markdown consistency, atomic writes, CLI overrides, mock/live API-key requirements, exit codes 0/1/2, and continued report writing after run failures.

• [x] Verify RED with uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q.

• [x] Implement aggregation into one typed BenchmarkReport model shared by JSON and Markdown.

• [x] Implement centralized redaction and 2 KiB error truncation before any rendered output.

• [x] Implement one atomically published report bundle containing matching report.json and report.md files.

• [x] Implement argparse with --config, --base-url, --model, repeatable --case/--mode, --runs, --output-dir, and --mock; apply CLI > JSON and environment-only API key.

• [x] Register agent-lab-benchmark = agent_lab.presentation.benchmark_cli:main and verify --help.

• [x] Run reporting/CLI/runner tests, then commit with feat: add benchmark reports and cli.

Task 4: Examples, documentation, and final verification

Files:

• Create: benchmarks/single-target.example.json
• Modify: .gitignore
• Modify: README.md
• Modify: docs/plans/todo-52-agent-benchmark-runner.md

• Modify: docs/plans/todos.md

• [x] Add a secret-free example covering both modes and all six cases; ignore outputs/benchmarks/.

• [x] Document .env, config copy/edit, mock command, live command, output fields, exit codes, fairness controls, and the limits of scripted/mock adapters.

• [x] Run agent-lab-benchmark --config benchmarks/single-target.example.json --mock --output-dir /tmp/agent-lab-benchmark and inspect both reports.

• [x] Run focused benchmark tests, uv run pytest, CLI help, app import, node --check, and git diff --check.

• [x] Perform specification review then code-quality review; add Todo 52.x rows for post-execution fixes.

• [x] Record evidence and Workflow Evolution findings, mark Todo 52 done, and commit with docs: close benchmark runner todo.

Final Evidence

Verified on 2026-07-14 (Asia/Shanghai):

• benchmark suite: 196 passed in 1.19s;
• full suite: 601 passed in 3.67s, with one existing Starlette/httpx deprecation warning;
• agent-lab-benchmark --help: exit 0;
• application import prints Agent Lab;
• node --check src/agent_lab/presentation/static/app.js: exit 0;
• git diff --check: exit 0;
• zero-network mock CLI: 12 attempted / 12 passed / 0 failed;
• final JSON: /tmp/agent-lab-benchmark-final-20260714/20260713T194850Z/report.json;
• final Markdown: /tmp/agent-lab-benchmark-final-20260714/20260713T194850Z/report.md;
• both parallel-mode runs report parallel_events_overlapped=true;
• both web-search runs report two model calls and one tool execution;
• no mock run has missing token usage.

Specification and code-quality reviews found no remaining Critical or Important issues after Todos 52.1-52.13. The remaining non-blocking risks are documented below.

Residual Risks and Limits

• No live provider request was run because no real benchmark target or credential was supplied; live latency, usage support, semantic adherence, and provider tool-stream compatibility still require operator verification.
• The example uses one run per case, so its 60 p95 warnings are expected and the mock percentile values are not suitable for ranking architectures.
• Atomic bundle publication uses a relative directory symlink and is verified on macOS; Windows symlink behavior is not verified.
• An interruption inside mkdtemp() may leave an empty hidden bundle directory. A close failure during a narrow consumer-pause cancellation window may be logged as a secondary async-generator exception. Neither case changes the primary cancellation result or exposes a mixed public report.

Workflow Evolution Update

The requirement loop exposed three durable workflow lessons:

• paired filesystem artifacts need one explicit atomic publication boundary; repeated two-file rollback fixes were replaced by a single report bundle;
• benchmark plans must define clock boundaries, missing-data semantics, and observable concurrency proof before implementation, not only metric names and event counts;
• parallel subagents need shared-contract integration checks when one task changes an output layout consumed by another task's tests.

These lessons are recorded in this task report and the benchmark design. No AGENTS.md change was made because the repository's existing PAOU, todo-first, TDD, per-todo commit, and post-review follow-up rules already caused the issues to be found and corrected; adding more global workflow rules would duplicate the current core process.