Преглед изворни кода

docs: plan benchmark output reservation fixes

Problem: Task 3 quality review found stale/overwrite races, argparse value leakage, and empty runner output reported as success.

Risk: Final-path placeholders must be cleaned precisely so failure recovery never deletes another process's reports.
zhenyu.hu пре 2 недеља
родитељ
комит
0475120a2a

+ 28 - 0
docs/plans/todo-52.7-benchmark-output-reservation-and-cardinality.md

@@ -0,0 +1,28 @@
+# Todo 52.7 Benchmark Output Reservation and Cardinality Plan
+
+**Status:** in_progress
+
+## Goal
+
+Eliminate remaining report publication races and prevent an incomplete benchmark execution from being reported as successful.
+
+## Scope
+
+- Reserve both final report paths with exclusive file creation before writing temporary content; never overwrite a path created by another process.
+- Remove lock/reservation sidecar files entirely; clean only placeholders/temporary/published files owned by the current attempt after failure.
+- Replace argparse error details with a generic safe message so malformed values cannot be echoed to stderr.
+- Require `len(results) == len(cases) * len(modes) * runs_per_case`; mismatches become redacted system errors in the report and exit code 2.
+
+## Files
+
+- Modify: `src/agent_lab/infrastructure/benchmark_reporting.py`
+- Modify: `src/agent_lab/presentation/benchmark_cli.py`
+- Modify: `tests/test_benchmark_reporting.py`
+- Modify: `tests/test_benchmark_cli.py`
+
+## Verification
+
+- `uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q`
+- concurrent reservation stress test
+- `uv run pytest`
+- `git diff --check`

+ 1 - 0
docs/plans/todos.md

@@ -107,3 +107,4 @@
 | 52.4 | done | `docs/plans/todo-52.4-benchmark-visible-ttft-and-call-counts.md` | Measure the first non-whitespace visible content and enforce each case's expected ChatAgent/fallback call counts. | Blank deltas cannot reduce visible TTFT; extra or missing model/fallback calls produce semantic failures even when timing and ledger agree. |
 | 52.5 | done | `docs/plans/todo-52.5-benchmark-ledger-event-identity.md` | Cross-check detected, batch-result, and tool-metric event identities and preserve attempted model counts on failed runs. | A successful result from the wrong event cannot pass; failed streams still report started ChatAgent/fallback attempts from timing records. |
 | 52.6 | in_progress | `docs/plans/todo-52.6-benchmark-report-auditability.md` | Complete report latency coverage, concurrent pair reservation, runner-level error auditability, and byte-bounded errors. | Reports aggregate model/tool latency, concurrent writers cannot mix pairs, catastrophic runner errors appear in report totals/details, zero samples warn, and errors stay within 2 KiB UTF-8. |
+| 52.7 | in_progress | `docs/plans/todo-52.7-benchmark-output-reservation-and-cardinality.md` | Prevent report overwrite/stale reservations, suppress argparse value echoes, and reject missing or extra benchmark results. | Final paths are atomically reserved without lock files, malformed CLI values are not echoed, and actual run cardinality must equal cases × modes × iterations. |