agent-lab/docs/plans/todo-52.5-benchmark-ledger-event-identity.md

Todo 52.5 Benchmark Ledger Event Identity Plan

Status: done

Goal

Prevent independent benchmark ledgers from agreeing on counts while referring to different events, and keep failed-run attempt counts truthful.

Scope

• Preserve event names from event_batch_results and tool_execution usage rows in BenchmarkRunResult.
• Require detected event names, batch-result event names, and tool-metric event names to equal the case's expected ordered event list.
• Keep normalized status checks paired with the batch event identity.
• Report ChatAgent/fallback attempt counts from timing records; continue checking timing counts against persisted ledger counts when persistence exists.
• For pre-ledger failures, retain timing-derived attempt counts instead of None/zero.

Files

• Modify: src/agent_lab/application/benchmark.py
• Modify: tests/test_benchmark_runner.py

Verification

• uv run pytest tests/test_benchmark_runner.py tests/test_benchmark_config.py -q
• uv run pytest
• git diff --check

Observe / Update

• Detected, batch-result, and tool-metric event names are preserved and independently matched to the expected ordered event list.
• Failed streams retain timing-derived model/fallback attempts and any usage already returned.
• Final Task 2 review approved the implementation; focused tests passed (115 passed) and the full suite passed (520 passed).