Преглед изворни кода

docs: close benchmark runner review fixes

Problem: Task 2 review fixes needed durable status and metric/identity evidence before reporting work.

Risk: None beyond the live-provider variability already recorded in the benchmark design.
zhenyu.hu пре 2 недеља
родитељ
комит
9cff699d0f

+ 7 - 1
docs/plans/todo-52.4-benchmark-visible-ttft-and-call-counts.md

@@ -1,6 +1,6 @@
 # Todo 52.4 Benchmark Visible TTFT and Call Counts Plan
 
-**Status:** in_progress
+**Status:** done
 
 ## Goal
 
@@ -24,3 +24,9 @@ Prevent benchmark runs from passing with hidden extra model cost or an artificia
 - `uv run pytest tests/test_benchmark_runner.py tests/test_benchmark_config.py -q`
 - `uv run pytest`
 - `git diff --check`
+
+## Observe / Update
+
+- Visible TTFT now ignores blank/whitespace deltas at both call and run boundaries.
+- Each case declares expected ChatAgent/fallback counts; ledger and timing attempts are checked against those expectations and each other.
+- Focused tests passed (`111 passed`) and the full suite passed at this checkpoint (`516 passed`).

+ 7 - 1
docs/plans/todo-52.5-benchmark-ledger-event-identity.md

@@ -1,6 +1,6 @@
 # Todo 52.5 Benchmark Ledger Event Identity Plan
 
-**Status:** in_progress
+**Status:** done
 
 ## Goal
 
@@ -24,3 +24,9 @@ Prevent independent benchmark ledgers from agreeing on counts while referring to
 - `uv run pytest tests/test_benchmark_runner.py tests/test_benchmark_config.py -q`
 - `uv run pytest`
 - `git diff --check`
+
+## Observe / Update
+
+- Detected, batch-result, and tool-metric event names are preserved and independently matched to the expected ordered event list.
+- Failed streams retain timing-derived model/fallback attempts and any usage already returned.
+- Final Task 2 review approved the implementation; focused tests passed (`115 passed`) and the full suite passed (`520 passed`).

+ 2 - 2
docs/plans/todos.md

@@ -104,5 +104,5 @@
 | 52.1 | done | `docs/plans/todo-52.1-benchmark-url-control-characters.md` | Reject non-printable/control characters in benchmark target URLs before constructing the HTTP client. | Strict config tests cover NUL, CR/LF, tabs, and other Unicode/control characters while valid HTTP(S) URLs remain accepted. |
 | 52.2 | done | `docs/plans/todo-52.2-benchmark-url-transport-compatibility.md` | Validate benchmark target hostname/URL compatibility with the production HTTP transport at the config boundary. | Invalid IDNA/port/transport URLs are rejected as config errors while valid Unicode hostnames accepted by the transport remain valid. |
 | 52.3 | done | `docs/plans/todo-52.3-benchmark-case-mode-single-source.md` | Make benchmark case IDs and invocation modes single-source enums shared by config, catalog, runner, and tests. | Adding or removing an enum member cannot leave config literals, exported choices, and catalog keys silently inconsistent. |
-| 52.4 | in_progress | `docs/plans/todo-52.4-benchmark-visible-ttft-and-call-counts.md` | Measure the first non-whitespace visible content and enforce each case's expected ChatAgent/fallback call counts. | Blank deltas cannot reduce visible TTFT; extra or missing model/fallback calls produce semantic failures even when timing and ledger agree. |
-| 52.5 | in_progress | `docs/plans/todo-52.5-benchmark-ledger-event-identity.md` | Cross-check detected, batch-result, and tool-metric event identities and preserve attempted model counts on failed runs. | A successful result from the wrong event cannot pass; failed streams still report started ChatAgent/fallback attempts from timing records. |
+| 52.4 | done | `docs/plans/todo-52.4-benchmark-visible-ttft-and-call-counts.md` | Measure the first non-whitespace visible content and enforce each case's expected ChatAgent/fallback call counts. | Blank deltas cannot reduce visible TTFT; extra or missing model/fallback calls produce semantic failures even when timing and ledger agree. |
+| 52.5 | done | `docs/plans/todo-52.5-benchmark-ledger-event-identity.md` | Cross-check detected, batch-result, and tool-metric event identities and preserve attempted model counts on failed runs. | A successful result from the wrong event cannot pass; failed streams still report started ChatAgent/fallback attempts from timing records. |