Quellcode durchsuchen

docs: close benchmark configuration follow-ups

Problem: Task 1 review fixes needed durable status and verification evidence before runner implementation.

Risk: The application benchmark module still directly uses httpx URL semantics; this recorded Minor does not affect current correctness.
zhenyu.hu vor 2 Wochen
Ursprung
Commit
9464c41c94

+ 6 - 1
docs/plans/todo-52.1-benchmark-url-control-characters.md

@@ -1,6 +1,6 @@
 # Todo 52.1 Benchmark URL Control Characters Plan
 
-**Status:** in_progress
+**Status:** done
 
 ## Goal
 
@@ -22,3 +22,8 @@ Keep malformed benchmark targets inside the configuration error boundary instead
 - `uv run pytest tests/test_benchmark_config.py tests/test_tool_invocation_comparison.py -q`
 - `uv run pytest`
 - `git diff --check`
+
+## Observe / Update
+
+- Control-category characters are rejected before transport construction while printable internationalized URLs remain covered.
+- Focused tests passed (`67 passed`) and the full suite passed at this checkpoint (`472 passed`).

+ 6 - 1
docs/plans/todo-52.2-benchmark-url-transport-compatibility.md

@@ -1,6 +1,6 @@
 # Todo 52.2 Benchmark URL Transport Compatibility Plan
 
-**Status:** in_progress
+**Status:** done
 
 ## Goal
 
@@ -22,3 +22,8 @@ Ensure every accepted benchmark base URL can be constructed by the production ht
 - `uv run pytest tests/test_benchmark_config.py tests/test_tool_invocation_comparison.py -q`
 - `uv run pytest`
 - `git diff --check`
+
+## Observe / Update
+
+- Config validation now uses the production httpx URL semantics for IDNA and port compatibility.
+- Focused tests passed (`72 passed`) and the full suite passed at this checkpoint (`477 passed`).

+ 6 - 1
docs/plans/todo-52.3-benchmark-case-mode-single-source.md

@@ -1,6 +1,6 @@
 # Todo 52.3 Benchmark Case and Mode Single Source Plan
 
-**Status:** in_progress
+**Status:** done
 
 ## Goal
 
@@ -24,3 +24,8 @@ Prevent benchmark configuration and the executable case catalog from drifting as
 - `uv run pytest tests/test_benchmark_config.py tests/test_tool_invocation_comparison.py -q`
 - `uv run pytest`
 - `git diff --check`
+
+## Observe / Update
+
+- `BenchmarkMode` and `BenchmarkCaseId` enums now drive config validation, exported choices, catalog keys, request construction, and the 12-run comparison matrix.
+- Specification and code-quality reviews approved the result; focused tests passed (`76 passed`) and the full suite passed (`481 passed`).

+ 3 - 3
docs/plans/todos.md

@@ -101,6 +101,6 @@
 | 51 | done | `docs/plans/todo-51-comparison-matrix-closeout.md` | Add repeatable comparison fixtures, update project documentation, and close the experimental implementation. | Both modes pass the same 12-run scenario matrix; focused (`231 passed`) and full (`417 passed`) tests, JS syntax, app import, and documentation checks pass. |
 | 51.1 | done | `docs/plans/todo-51.1-first-reply-and-protocol-matrix.md` | Require fast visible first replies for event scenarios and prove each mode uses its intended capability protocol surface. | Matrix asserts first reply ordering, no unintended second reply, dual compact event catalog/no provider tools, direct enabled schemas/no generated event catalog, and correct event source. |
 | 52 | in_progress | `docs/plans/todo-52-agent-benchmark-runner.md` | Add a configurable in-process benchmark CLI with deterministic mock cases and live OpenAI-compatible execution. | Mock/live runs compare both modes, measure provider/visible TTFT, wall time, tokens and event correctness, write secret-safe JSON/Markdown reports, and use stable exit codes. |
-| 52.1 | in_progress | `docs/plans/todo-52.1-benchmark-url-control-characters.md` | Reject non-printable/control characters in benchmark target URLs before constructing the HTTP client. | Strict config tests cover NUL, CR/LF, tabs, and other Unicode/control characters while valid HTTP(S) URLs remain accepted. |
-| 52.2 | in_progress | `docs/plans/todo-52.2-benchmark-url-transport-compatibility.md` | Validate benchmark target hostname/URL compatibility with the production HTTP transport at the config boundary. | Invalid IDNA/port/transport URLs are rejected as config errors while valid Unicode hostnames accepted by the transport remain valid. |
-| 52.3 | in_progress | `docs/plans/todo-52.3-benchmark-case-mode-single-source.md` | Make benchmark case IDs and invocation modes single-source enums shared by config, catalog, runner, and tests. | Adding or removing an enum member cannot leave config literals, exported choices, and catalog keys silently inconsistent. |
+| 52.1 | done | `docs/plans/todo-52.1-benchmark-url-control-characters.md` | Reject non-printable/control characters in benchmark target URLs before constructing the HTTP client. | Strict config tests cover NUL, CR/LF, tabs, and other Unicode/control characters while valid HTTP(S) URLs remain accepted. |
+| 52.2 | done | `docs/plans/todo-52.2-benchmark-url-transport-compatibility.md` | Validate benchmark target hostname/URL compatibility with the production HTTP transport at the config boundary. | Invalid IDNA/port/transport URLs are rejected as config errors while valid Unicode hostnames accepted by the transport remain valid. |
+| 52.3 | done | `docs/plans/todo-52.3-benchmark-case-mode-single-source.md` | Make benchmark case IDs and invocation modes single-source enums shared by config, catalog, runner, and tests. | Adding or removing an enum member cannot leave config literals, exported choices, and catalog keys silently inconsistent. |