Jelajahi Sumber

docs: plan atomic benchmark report bundles

Problem: Publishing JSON and Markdown as two independent final paths leaves an unavoidable check-then-unlink rollback race and can expose mixed report pairs.

Risk: The follow-up changes the report output layout to an atomically published bundle and must preserve CLI paths, report contents, and concurrent-writer behavior.
zhenyu.hu 2 minggu lalu
induk
melakukan
cf26628654

+ 37 - 0
docs/plans/todo-52.9-atomic-report-bundles-and-audit-parity.md

@@ -0,0 +1,37 @@
+# Todo 52.9 Atomic Report Bundles and Audit Parity Plan
+
+**Status:** in_progress
+
+## Goal
+
+Replace the fundamentally non-transactional two-path report publication with one atomic bundle publication boundary, and close the remaining report/CLI audit gaps found after Todo 52.8.
+
+## Root Cause
+
+Publishing flat JSON and Markdown paths requires two independent filesystem operations. If the second operation conflicts, rolling back the first requires a check-then-unlink sequence that cannot atomically prove pathname ownership. A competing process can replace the first path between the inode check and `unlink()`, so the writer may delete another process's file or return a mixed pair.
+
+## Scope
+
+- Fully write and `fsync` `report.json` and `report.md` inside one private, uniquely named bundle directory.
+- Publish the complete bundle through one atomic, no-overwrite directory symlink named from the timestamp; a collision chooses the next numeric suffix without deleting or replacing the competitor.
+- Return the two report paths through the published bundle path and leave no flat two-file rollback path.
+- Clean only unpublished private bundle state. Never inspect-then-delete a published competitor-controlled pathname.
+- Treat mutation after successful publication as out of scope: a process with write access to the output directory can replace any published filesystem entry after the command returns.
+- Render runtime errors and semantic failures together in Markdown when both exist, preserving the facts present in JSON.
+- Map summary/error stream failures to exit code `2` without exposing secrets or confusing them with benchmark sample failures.
+
+## Files
+
+- Modify: `src/agent_lab/infrastructure/benchmark_reporting.py`
+- Modify: `src/agent_lab/presentation/benchmark_cli.py`
+- Modify: `tests/test_benchmark_reporting.py`
+- Modify: `tests/test_benchmark_cli.py`
+
+## Verification
+
+- Reproduce the Todo 52.8 inode-check-to-unlink race before implementation.
+- `uv run pytest tests/test_benchmark_reporting.py tests/test_benchmark_cli.py -q`
+- concurrent process/thread bundle publication and collision stress tests
+- `uv run pytest`
+- `git diff --check`
+

+ 1 - 0
docs/plans/todos.md

@@ -109,3 +109,4 @@
 | 52.6 | in_progress | `docs/plans/todo-52.6-benchmark-report-auditability.md` | Complete report latency coverage, concurrent pair reservation, runner-level error auditability, and byte-bounded errors. | Reports aggregate model/tool latency, concurrent writers cannot mix pairs, catastrophic runner errors appear in report totals/details, zero samples warn, and errors stay within 2 KiB UTF-8. |
 | 52.7 | in_progress | `docs/plans/todo-52.7-benchmark-output-reservation-and-cardinality.md` | Prevent report overwrite/stale reservations, suppress argparse value echoes, and reject missing or extra benchmark results. | Final paths are atomically reserved without lock files, malformed CLI values are not echoed, and actual run cardinality must equal cases × modes × iterations. |
 | 52.8 | in_progress | `docs/plans/todo-52.8-benchmark-no-overwrite-publication.md` | Publish completed report files with atomic no-overwrite hard links instead of check-then-replace ownership logic. | A destination created at any point wins without being overwritten; the writer retries another suffix or fails cleanly with no mixed pair. |
+| 52.9 | in_progress | `docs/plans/todo-52.9-atomic-report-bundles-and-audit-parity.md` | Replace two-path report publication with one atomic bundle boundary and close report/CLI audit parity gaps. | Concurrent writers publish complete bundles without rollback deletion races; Markdown preserves all failure facts and output errors map to exit code 2. |