|
|
@@ -0,0 +1,30 @@
|
|
|
+# Todo 52.10 Benchmark Bundle CLI Contract Plan
|
|
|
+
|
|
|
+**Status:** in_progress
|
|
|
+
|
|
|
+## Goal
|
|
|
+
|
|
|
+Align CLI integration tests and report discovery with the atomic bundle layout introduced by Todo 52.9.
|
|
|
+
|
|
|
+## Observation
|
|
|
+
|
|
|
+The CLI still returns the report paths supplied by `write_benchmark_report()`, but four tests discover reports independently with root-level `*.json`/`*.md` globs or attempt to read every output-directory entry as a file. The new layout intentionally exposes a timestamped bundle symlink containing `report.json` and `report.md`, plus a private backing directory.
|
|
|
+
|
|
|
+## Scope
|
|
|
+
|
|
|
+- Update CLI tests to use the exact `json_report` and `markdown_report` paths printed by the command, or inspect the single published bundle symlink.
|
|
|
+- Do not inspect private dot-prefixed bundle backing directories as public reports.
|
|
|
+- Preserve assertions for CLI overrides, exit codes, cardinality system errors, report redaction, and matching JSON/Markdown contents.
|
|
|
+- Avoid adding compatibility copies at the output root because that would restore the two-path publication race fixed by Todo 52.9.
|
|
|
+
|
|
|
+## Files
|
|
|
+
|
|
|
+- Modify: `tests/test_benchmark_cli.py`
|
|
|
+
|
|
|
+## Verification
|
|
|
+
|
|
|
+- `uv run pytest tests/test_benchmark_cli.py -q`
|
|
|
+- `uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q`
|
|
|
+- `uv run pytest`
|
|
|
+- `git diff --check`
|
|
|
+
|