agent-lab/docs/plans/todo-52.10-benchmark-bundle-cli-contract.md

Todo 52.10 Benchmark Bundle CLI Contract Plan

Status: in_progress

Goal

Align CLI integration tests and report discovery with the atomic bundle layout introduced by Todo 52.9.

Observation

The CLI still returns the report paths supplied by write_benchmark_report(), but four tests discover reports independently with root-level *.json/*.md globs or attempt to read every output-directory entry as a file. The new layout intentionally exposes a timestamped bundle symlink containing report.json and report.md, plus a private backing directory.

Scope

• Update CLI tests to use the exact json_report and markdown_report paths printed by the command, or inspect the single published bundle symlink.
• Do not inspect private dot-prefixed bundle backing directories as public reports.
• Preserve assertions for CLI overrides, exit codes, cardinality system errors, report redaction, and matching JSON/Markdown contents.
• Avoid adding compatibility copies at the output root because that would restore the two-path publication race fixed by Todo 52.9.

Files

• Modify: tests/test_benchmark_cli.py

Verification

• uv run pytest tests/test_benchmark_cli.py -q
• uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q
• uv run pytest
• git diff --check