todo-51.1-first-reply-and-protocol-matrix.md 1.1 KB

Todo 51.1 First Reply and Protocol Matrix Plan

Status: in_progress

Goal

Make the closeout matrix enforce the experiment's primary latency contract and the intended mode-specific protocol surfaces.

Scope

  • Add a ChatAgent message_delta before events for volume, schedule, terminate, web search, and parallel volume/schedule scenarios.
  • Assert first visible reply precedes the first tool result; silent/terminate flows do not add an unintended second model answer; schedule/parallel add exactly one deterministic template after execution.
  • Record each scripted model request.
  • For dual_agent, assert the initial ChatAgent request has no provider tools and includes the compact enabled event catalog.
  • For chat_agent_tools, assert the initial request contains exactly the enabled provider schemas and no generated event catalog.
  • Assert detected event audit source is text_event for dual mode and provider_resolved for direct mode.

Files

  • Modify: tests/test_tool_invocation_comparison.py

Verification

  • uv run pytest tests/test_tool_invocation_comparison.py -q
  • repeat parallel cases
  • uv run pytest
  • git diff --check