agent-lab/docs/plans/todo-51.1-first-reply-and-protocol-matrix.md

Todo 51.1 First Reply and Protocol Matrix Plan

Status: done

Goal

Make the closeout matrix enforce the experiment's primary latency contract and the intended mode-specific protocol surfaces.

Scope

• Add a ChatAgent message_delta before events for volume, schedule, terminate, web search, and parallel volume/schedule scenarios.
• Assert first visible reply precedes the first tool result; silent/terminate flows do not add an unintended second model answer; schedule/parallel add exactly one deterministic template after execution.
• Record each scripted model request.
• For dual_agent, assert the initial ChatAgent request has no provider tools and includes the compact enabled event catalog.
• For chat_agent_tools, assert the initial request contains exactly the enabled provider schemas and no generated event catalog.
• Assert detected event audit source is text_event for dual mode and provider_resolved for direct mode.

Files

• Modify: tests/test_tool_invocation_comparison.py

Verification

• uv run pytest tests/test_tool_invocation_comparison.py -q
• repeat parallel cases
• uv run pytest
• git diff --check

Observe / Update

• Side-effect scenarios now stream a natural ChatAgent acknowledgement/farewell before event completion.
• Matrix request capture proves dual mode sends no provider tools and injects exactly the enabled compact event catalog; direct mode sends exactly the enabled schemas and no generated catalog.
• Detected event audits report text_event for dual mode and provider_resolved for direct mode.
• Matrix passed (12 passed), parallel scenarios passed 20 repeated runs, and the full suite passed (417 passed).