Selaa lähdekoodia

docs: plan benchmark case mode single source

Problem: Task 1 quality review found benchmark modes, case literals, exported choices, and catalog keys could drift independently.

Risk: Enum migration must preserve JSON string compatibility and existing runtime request values.
zhenyu.hu 2 viikkoa sitten
vanhempi
commit
9ed4d56ceb
2 muutettua tiedostoa jossa 27 lisäystä ja 0 poistoa
  1. 26 0
      docs/plans/todo-52.3-benchmark-case-mode-single-source.md
  2. 1 0
      docs/plans/todos.md

+ 26 - 0
docs/plans/todo-52.3-benchmark-case-mode-single-source.md

@@ -0,0 +1,26 @@
+# Todo 52.3 Benchmark Case and Mode Single Source Plan
+
+**Status:** in_progress
+
+## Goal
+
+Prevent benchmark configuration and the executable case catalog from drifting as new cases or modes are added.
+
+## Scope
+
+- Replace separate `Literal` declarations with `StrEnum` definitions for case IDs and invocation modes.
+- Derive exported ordered choice tuples from the enums.
+- Type catalog keys and config lists with those enums while preserving JSON string serialization and existing runtime request values.
+- Assert every case enum has exactly one catalog entry.
+
+## Files
+
+- Modify: `src/agent_lab/application/benchmark.py`
+- Modify: `tests/test_benchmark_config.py`
+- Modify: `tests/test_tool_invocation_comparison.py`
+
+## Verification
+
+- `uv run pytest tests/test_benchmark_config.py tests/test_tool_invocation_comparison.py -q`
+- `uv run pytest`
+- `git diff --check`

+ 1 - 0
docs/plans/todos.md

@@ -103,3 +103,4 @@
 | 52 | in_progress | `docs/plans/todo-52-agent-benchmark-runner.md` | Add a configurable in-process benchmark CLI with deterministic mock cases and live OpenAI-compatible execution. | Mock/live runs compare both modes, measure provider/visible TTFT, wall time, tokens and event correctness, write secret-safe JSON/Markdown reports, and use stable exit codes. |
 | 52.1 | in_progress | `docs/plans/todo-52.1-benchmark-url-control-characters.md` | Reject non-printable/control characters in benchmark target URLs before constructing the HTTP client. | Strict config tests cover NUL, CR/LF, tabs, and other Unicode/control characters while valid HTTP(S) URLs remain accepted. |
 | 52.2 | in_progress | `docs/plans/todo-52.2-benchmark-url-transport-compatibility.md` | Validate benchmark target hostname/URL compatibility with the production HTTP transport at the config boundary. | Invalid IDNA/port/transport URLs are rejected as config errors while valid Unicode hostnames accepted by the transport remain valid. |
+| 52.3 | in_progress | `docs/plans/todo-52.3-benchmark-case-mode-single-source.md` | Make benchmark case IDs and invocation modes single-source enums shared by config, catalog, runner, and tests. | Adding or removing an enum member cannot leave config literals, exported choices, and catalog keys silently inconsistent. |