agent-lab/docs/plans/todo-51-comparison-matrix-closeout.md

Todo 51 Comparison Matrix Closeout Plan

Status: done

Goal

Close the service experiment with one repeatable behavior matrix, current operational documentation, explicit comparison controls, and bounded conclusions.

Scenario Matrix

Run the same six business scenarios in both dual_agent and chat_agent_tools modes (12 parameterized runs):

1. ordinary chat with all built-ins enabled and no event;
2. absolute device-volume adjustment with silent success;
3. fixed RFC3339 schedule creation with one deterministic confirmation;
4. web search with first answer, tool result, then exactly one second answer;
5. session termination with exactly one farewell and terminal completion;
6. concurrent volume plus schedule, bounded at two workers, preserving input result order and one schedule template.

The matrix compares shared user-visible semantics and normalized business results. It deliberately allows provider transcript and event-selection protocol differences.

Fair Comparison Controls

• Same model parameters, enabled event set, user inputs, output budget, registry, plugin schemas, handlers, event budget, concurrency, and timeouts.
• Direct-mode provider tool protocol tokens remain counted.
• Token totals include only real model calls; tool rows have zero tokens.
• turn_wall_time_ms is the only turn/session wall-clock total; overlapping model/tool spans are not summed.
• Deduplicated/replayed event clones count as one logical tool execution.
• Web search requires two ChatAgent answers in both modes.
• Provider TTFT and visible-text TTFT remain distinct.

Documentation Scope

• Rewrite README.md as the operational entry for architecture, built-ins, modes, configuration, UI/WebSocket/API use, metrics, verification, and current adapter limitations.
• Calibrate docs/plans/2026-07-12-chat-event-agent-production-events-design.md to implemented namespaced event names and link this closeout; keep it as the architecture fact source.
• Record matrix evidence, conclusions, and remaining risks in this file.
• Update docs/plans/todos.md only when final verification passes.

Files

• Add: tests/test_tool_invocation_comparison.py
• Modify: README.md
• Modify: docs/plans/2026-07-12-chat-event-agent-production-events-design.md
• Modify: this file and docs/plans/todos.md during final Update.

Requirement Execution Loop

Plan

1. Add the 12-run RED comparison matrix with small local scripted clients and injected production plugin ports.
2. Fix only genuine shared-contract gaps exposed by the matrix; use Todo 51.x for any post-execution repair.
3. Update README and architecture design names/status from the verified implementation.
4. Run matrix, focused, full, JS syntax, app import, and diff checks.
5. Perform specification and quality review, then record evidence and bounded experiment conclusions.

Verification

• uv run pytest tests/test_tool_invocation_comparison.py -q
• uv run pytest tests/test_debug_runtime.py tests/test_builtin_event_plugins.py tests/test_sqlite_store.py tests/test_websocket_api.py -q
• uv run pytest
• node --check src/agent_lab/presentation/static/app.js
• uv run python -c "from agent_lab.main import app; print(app.title)"
• git diff --check

Observe

• The shared matrix runs 6 scenarios in both modes (12 passed): ordinary chat, volume, schedule, web search, termination, and parallel volume plus schedule.
• Review required Todo 51.1 because the first matrix did not enforce fast first replies or inspect the actual mode-specific model request surface.
• After Todo 51.1, each event scenario proves the first ChatAgent delta precedes tool completion; web search has exactly two model answers; schedule/parallel add one deterministic template; silent/terminate flows add no unintended second model answer.
• dual_agent proves compact generated event catalog plus no provider tools; chat_agent_tools proves exact enabled schemas plus no generated catalog. Audit source is mode-correct.
• Parallel cases require both handlers to start, release in reverse completion order, and still emit tool results in accepted input order.

Verification Evidence

• Matrix: 12 passed in 0.22s.
• Focused runtime/plugin/storage/WebSocket regression: 231 passed, one existing Starlette deprecation warning.
• Full suite: 417 passed, one existing Starlette deprecation warning.
• node --check src/agent_lab/presentation/static/app.js: passed.
• App import printed Agent Lab.
• git diff --check: passed.

Experiment Conclusions

Confirmed by this implementation and matrix:

• one generic event kernel and flat built-in plugin registry serve both invocation modes;
• both modes preserve the same normalized event execution and user-visible result-policy contract;
• fast first replies, web-search two-answer ordering, multi-event parallelism, terminal behavior, session restoration, and comparable metric accounting have repeatable regression coverage;
• token totals, logical tool counts, fallback rows, and turn wall time are separated so the two architectures can be measured without known double-counting.

Not established by this closeout:

• which mode has lower real-provider TTFT, turn latency, token cost, or monetary cost;
• which mode produces better natural-language quality or event accuracy on live stochastic models;
• real device, calendar, dialogue termination, and internet-search reliability.

The current matrix uses scripted model streams and injected/default in-memory adapters. A production decision still requires paired live-provider and service-adapter benchmarks using fresh databases and the same comparison controls.

Remaining Risks

• Some providers may suppress visible content when emitting native tool calls, weakening direct mode's first-reply behavior.
• Fallback frequency and schema/prompt growth depend on the real capability set and user distribution.
• Model randomness, provider usage reporting, and real search latency can dominate local deterministic results.
• Reusing historical SQLite sessions can contaminate paired comparisons; benchmark runs should use isolated databases.
• Monetary cost is not stored because it depends on provider/model pricing; estimate it from model call tokens outside the runtime.

Workflow Evolution Update

• The requirement/todo loop, isolated worktree, per-todo commits, TDD, and follow-up x.x rows successfully contained review findings without rewriting completed todos.
• Subagent model overrides and the default user-level uv cache were environment-specific friction. The main session used inherited subagent routing, explicit verification, and controlled Git commits. No repository workflow rule was added because these are not stable project requirements.