فهرست منبع

docs: plan benchmark bundle cli integration

Problem: CLI integration tests still discover root-level report files after Todo 52.9 moved publication to one atomic bundle.

Risk: Updating tests must not hide regressions in report paths, redaction, cardinality errors, or exit-code behavior.
zhenyu.hu 2 هفته پیش
والد
کامیت
e445f2d52e
2فایلهای تغییر یافته به همراه31 افزوده شده و 0 حذف شده
  1. 30 0
      docs/plans/todo-52.10-benchmark-bundle-cli-contract.md
  2. 1 0
      docs/plans/todos.md

+ 30 - 0
docs/plans/todo-52.10-benchmark-bundle-cli-contract.md

@@ -0,0 +1,30 @@
+# Todo 52.10 Benchmark Bundle CLI Contract Plan
+
+**Status:** in_progress
+
+## Goal
+
+Align CLI integration tests and report discovery with the atomic bundle layout introduced by Todo 52.9.
+
+## Observation
+
+The CLI still returns the report paths supplied by `write_benchmark_report()`, but four tests discover reports independently with root-level `*.json`/`*.md` globs or attempt to read every output-directory entry as a file. The new layout intentionally exposes a timestamped bundle symlink containing `report.json` and `report.md`, plus a private backing directory.
+
+## Scope
+
+- Update CLI tests to use the exact `json_report` and `markdown_report` paths printed by the command, or inspect the single published bundle symlink.
+- Do not inspect private dot-prefixed bundle backing directories as public reports.
+- Preserve assertions for CLI overrides, exit codes, cardinality system errors, report redaction, and matching JSON/Markdown contents.
+- Avoid adding compatibility copies at the output root because that would restore the two-path publication race fixed by Todo 52.9.
+
+## Files
+
+- Modify: `tests/test_benchmark_cli.py`
+
+## Verification
+
+- `uv run pytest tests/test_benchmark_cli.py -q`
+- `uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q`
+- `uv run pytest`
+- `git diff --check`
+

+ 1 - 0
docs/plans/todos.md

@@ -110,3 +110,4 @@
 | 52.7 | in_progress | `docs/plans/todo-52.7-benchmark-output-reservation-and-cardinality.md` | Prevent report overwrite/stale reservations, suppress argparse value echoes, and reject missing or extra benchmark results. | Final paths are atomically reserved without lock files, malformed CLI values are not echoed, and actual run cardinality must equal cases × modes × iterations. |
 | 52.8 | in_progress | `docs/plans/todo-52.8-benchmark-no-overwrite-publication.md` | Publish completed report files with atomic no-overwrite hard links instead of check-then-replace ownership logic. | A destination created at any point wins without being overwritten; the writer retries another suffix or fails cleanly with no mixed pair. |
 | 52.9 | in_progress | `docs/plans/todo-52.9-atomic-report-bundles-and-audit-parity.md` | Replace two-path report publication with one atomic bundle boundary and close report/CLI audit parity gaps. | Concurrent writers publish complete bundles without rollback deletion races; Markdown preserves all failure facts and output errors map to exit code 2. |
+| 52.10 | in_progress | `docs/plans/todo-52.10-benchmark-bundle-cli-contract.md` | Align CLI report discovery tests with the atomically published bundle layout. | CLI tests consume the reported bundle paths without restoring root-level compatibility copies or weakening redaction, exit-code, and cardinality checks. |