# Todo 51 Comparison Matrix Closeout Plan

**Status:** done

## Goal

Close the service experiment with one repeatable behavior matrix, current operational documentation, explicit comparison controls, and bounded conclusions.

## Scenario Matrix

Run the same six business scenarios in both `dual_agent` and `chat_agent_tools` modes (12 parameterized runs):

1. ordinary chat with all built-ins enabled and no event;
2. absolute device-volume adjustment with silent success;
3. fixed RFC3339 schedule creation with one deterministic confirmation;
4. web search with first answer, tool result, then exactly one second answer;
5. session termination with exactly one farewell and terminal completion;
6. concurrent volume plus schedule, bounded at two workers, preserving input result order and one schedule template.

The matrix compares shared user-visible semantics and normalized business results. It deliberately allows provider transcript and event-selection protocol differences.

## Fair Comparison Controls

- Same model parameters, enabled event set, user inputs, output budget, registry, plugin schemas, handlers, event budget, concurrency, and timeouts.
- Direct-mode provider tool protocol tokens remain counted.
- Token totals include only real model calls; tool rows have zero tokens.
- `turn_wall_time_ms` is the only turn/session wall-clock total; overlapping model/tool spans are not summed.
- Deduplicated/replayed event clones count as one logical tool execution.
- Web search requires two ChatAgent answers in both modes.
- Provider TTFT and visible-text TTFT remain distinct.

## Documentation Scope

- Rewrite `README.md` as the operational entry for architecture, built-ins, modes, configuration, UI/WebSocket/API use, metrics, verification, and current adapter limitations.
- Calibrate `docs/plans/2026-07-12-chat-event-agent-production-events-design.md` to implemented namespaced event names and link this closeout; keep it as the architecture fact source.
- Record matrix evidence, conclusions, and remaining risks in this file.
- Update `docs/plans/todos.md` only when final verification passes.

## Files

- Add: `tests/test_tool_invocation_comparison.py`
- Modify: `README.md`
- Modify: `docs/plans/2026-07-12-chat-event-agent-production-events-design.md`
- Modify: this file and `docs/plans/todos.md` during final Update.

## Requirement Execution Loop

### Plan

1. Add the 12-run RED comparison matrix with small local scripted clients and injected production plugin ports.
2. Fix only genuine shared-contract gaps exposed by the matrix; use Todo 51.x for any post-execution repair.
3. Update README and architecture design names/status from the verified implementation.
4. Run matrix, focused, full, JS syntax, app import, and diff checks.
5. Perform specification and quality review, then record evidence and bounded experiment conclusions.

## Verification

- `uv run pytest tests/test_tool_invocation_comparison.py -q`
- `uv run pytest tests/test_debug_runtime.py tests/test_builtin_event_plugins.py tests/test_sqlite_store.py tests/test_websocket_api.py -q`
- `uv run pytest`
- `node --check src/agent_lab/presentation/static/app.js`
- `uv run python -c "from agent_lab.main import app; print(app.title)"`
- `git diff --check`

## Observe

- The shared matrix runs 6 scenarios in both modes (`12 passed`): ordinary chat, volume, schedule, web search, termination, and parallel volume plus schedule.
- Review required Todo 51.1 because the first matrix did not enforce fast first replies or inspect the actual mode-specific model request surface.
- After Todo 51.1, each event scenario proves the first ChatAgent delta precedes tool completion; web search has exactly two model answers; schedule/parallel add one deterministic template; silent/terminate flows add no unintended second model answer.
- `dual_agent` proves compact generated event catalog plus no provider tools; `chat_agent_tools` proves exact enabled schemas plus no generated catalog. Audit source is mode-correct.
- Parallel cases require both handlers to start, release in reverse completion order, and still emit tool results in accepted input order.

## Verification Evidence

- Matrix: `12 passed in 0.22s`.
- Focused runtime/plugin/storage/WebSocket regression: `231 passed`, one existing Starlette deprecation warning.
- Full suite: `417 passed`, one existing Starlette deprecation warning.
- `node --check src/agent_lab/presentation/static/app.js`: passed.
- App import printed `Agent Lab`.
- `git diff --check`: passed.

## Experiment Conclusions

Confirmed by this implementation and matrix:

- one generic event kernel and flat built-in plugin registry serve both invocation modes;
- both modes preserve the same normalized event execution and user-visible result-policy contract;
- fast first replies, web-search two-answer ordering, multi-event parallelism, terminal behavior, session restoration, and comparable metric accounting have repeatable regression coverage;
- token totals, logical tool counts, fallback rows, and turn wall time are separated so the two architectures can be measured without known double-counting.

Not established by this closeout:

- which mode has lower real-provider TTFT, turn latency, token cost, or monetary cost;
- which mode produces better natural-language quality or event accuracy on live stochastic models;
- real device, calendar, dialogue termination, and internet-search reliability.

The current matrix uses scripted model streams and injected/default in-memory adapters. A production decision still requires paired live-provider and service-adapter benchmarks using fresh databases and the same comparison controls.

## Remaining Risks

- Some providers may suppress visible content when emitting native tool calls, weakening direct mode's first-reply behavior.
- Fallback frequency and schema/prompt growth depend on the real capability set and user distribution.
- Model randomness, provider usage reporting, and real search latency can dominate local deterministic results.
- Reusing historical SQLite sessions can contaminate paired comparisons; benchmark runs should use isolated databases.
- Monetary cost is not stored because it depends on provider/model pricing; estimate it from model call tokens outside the runtime.

## Workflow Evolution Update

- The requirement/todo loop, isolated worktree, per-todo commits, TDD, and follow-up `x.x` rows successfully contained review findings without rewriting completed todos.
- Subagent model overrides and the default user-level uv cache were environment-specific friction. The main session used inherited subagent routing, explicit verification, and controlled Git commits. No repository workflow rule was added because these are not stable project requirements.