agent-lab/docs/plans/todo-52.5-benchmark-ledger-event-identity.md

Todo 52.5 Benchmark Ledger Event Identity Plan

Status: in_progress

Goal

Prevent independent benchmark ledgers from agreeing on counts while referring to different events, and keep failed-run attempt counts truthful.

Scope

• Preserve event names from event_batch_results and tool_execution usage rows in BenchmarkRunResult.
• Require detected event names, batch-result event names, and tool-metric event names to equal the case's expected ordered event list.
• Keep normalized status checks paired with the batch event identity.
• Report ChatAgent/fallback attempt counts from timing records; continue checking timing counts against persisted ledger counts when persistence exists.
• For pre-ledger failures, retain timing-derived attempt counts instead of None/zero.

Files

• Modify: src/agent_lab/application/benchmark.py
• Modify: tests/test_benchmark_runner.py

Verification

• uv run pytest tests/test_benchmark_runner.py tests/test_benchmark_config.py -q
• uv run pytest
• git diff --check