todo-52.3-benchmark-case-mode-single-source.md 870 B

Todo 52.3 Benchmark Case and Mode Single Source Plan

Status: in_progress

Goal

Prevent benchmark configuration and the executable case catalog from drifting as new cases or modes are added.

Scope

  • Replace separate Literal declarations with StrEnum definitions for case IDs and invocation modes.
  • Derive exported ordered choice tuples from the enums.
  • Type catalog keys and config lists with those enums while preserving JSON string serialization and existing runtime request values.
  • Assert every case enum has exactly one catalog entry.

Files

  • Modify: src/agent_lab/application/benchmark.py
  • Modify: tests/test_benchmark_config.py
  • Modify: tests/test_tool_invocation_comparison.py

Verification

  • uv run pytest tests/test_benchmark_config.py tests/test_tool_invocation_comparison.py -q
  • uv run pytest
  • git diff --check