agent-lab/docs/plans/todo-52.4-benchmark-visible-ttft-and-call-counts.md

Todo 52.4 Benchmark Visible TTFT and Call Counts Plan

Status: done

Goal

Prevent benchmark runs from passing with hidden extra model cost or an artificially early visible-response timestamp.

Scope

• Record call-level and run-level visible TTFT only for a message_delta containing non-whitespace content.
• Add expected ChatAgent call count and expected fallback count to each benchmark case expectation.
• Compare both persisted ledger counts and timing-wrapper counts against the case expectation.
• Keep the six current cases at zero fallback; web search expects two ChatAgent calls and all other cases expect one.

Files

• Modify: src/agent_lab/application/benchmark.py
• Modify: tests/test_benchmark_config.py
• Modify: tests/test_benchmark_runner.py

Verification

• uv run pytest tests/test_benchmark_runner.py tests/test_benchmark_config.py -q
• uv run pytest
• git diff --check

Observe / Update

• Visible TTFT now ignores blank/whitespace deltas at both call and run boundaries.
• Each case declares expected ChatAgent/fallback counts; ledger and timing attempts are checked against those expectations and each other.
• Focused tests passed (111 passed) and the full suite passed at this checkpoint (516 passed).