docs: plan invocation comparison closeout

Problem: The experiment needs one repeatable cross-mode behavior matrix and current operational documentation before conclusions can be drawn.

Risk: Scripted clients and mock adapters validate contract parity but cannot establish real-provider latency, cost, or answer-quality winners.

docs/plans/todo-51-comparison-matrix-closeout.md

@@ -0,0 +1,63 @@
+# Todo 51 Comparison Matrix Closeout Plan
+
+**Status:** in_progress
+
+## Goal
+
+Close the service experiment with one repeatable behavior matrix, current operational documentation, explicit comparison controls, and bounded conclusions.
+
+## Scenario Matrix
+
+Run the same six business scenarios in both `dual_agent` and `chat_agent_tools` modes (12 parameterized runs):
+
+1. ordinary chat with all built-ins enabled and no event;
+2. absolute device-volume adjustment with silent success;
+3. fixed RFC3339 schedule creation with one deterministic confirmation;
+4. web search with first answer, tool result, then exactly one second answer;
+5. session termination with exactly one farewell and terminal completion;
+6. concurrent volume plus schedule, bounded at two workers, preserving input result order and one schedule template.
+
+The matrix compares shared user-visible semantics and normalized business results. It deliberately allows provider transcript and event-selection protocol differences.
+
+## Fair Comparison Controls
+
+- Same model parameters, enabled event set, user inputs, output budget, registry, plugin schemas, handlers, event budget, concurrency, and timeouts.
+- Direct-mode provider tool protocol tokens remain counted.
+- Token totals include only real model calls; tool rows have zero tokens.
+- `turn_wall_time_ms` is the only turn/session wall-clock total; overlapping model/tool spans are not summed.
+- Deduplicated/replayed event clones count as one logical tool execution.
+- Web search requires two ChatAgent answers in both modes.
+- Provider TTFT and visible-text TTFT remain distinct.
+
+## Documentation Scope
+
+- Rewrite `README.md` as the operational entry for architecture, built-ins, modes, configuration, UI/WebSocket/API use, metrics, verification, and current adapter limitations.
+- Calibrate `docs/plans/2026-07-12-chat-event-agent-production-events-design.md` to implemented namespaced event names and link this closeout; keep it as the architecture fact source.
+- Record matrix evidence, conclusions, and remaining risks in this file.
+- Update `docs/plans/todos.md` only when final verification passes.
+
+## Files
+
+- Add: `tests/test_tool_invocation_comparison.py`
+- Modify: `README.md`
+- Modify: `docs/plans/2026-07-12-chat-event-agent-production-events-design.md`
+- Modify: this file and `docs/plans/todos.md` during final Update.
+
+## Requirement Execution Loop
+
+### Plan
+
+1. Add the 12-run RED comparison matrix with small local scripted clients and injected production plugin ports.
+2. Fix only genuine shared-contract gaps exposed by the matrix; use Todo 51.x for any post-execution repair.
+3. Update README and architecture design names/status from the verified implementation.
+4. Run matrix, focused, full, JS syntax, app import, and diff checks.
+5. Perform specification and quality review, then record evidence and bounded experiment conclusions.
+
+## Verification
+
+- `uv run pytest tests/test_tool_invocation_comparison.py -q`
+- `uv run pytest tests/test_debug_runtime.py tests/test_builtin_event_plugins.py tests/test_sqlite_store.py tests/test_websocket_api.py -q`
+- `uv run pytest`
+- `node --check src/agent_lab/presentation/static/app.js`
+- `uv run python -c "from agent_lab.main import app; print(app.title)"`
+- `git diff --check`

docs/plans/todos.md

@@ -98,4 +98,4 @@
 | 49.3 | done | `docs/plans/todo-49.3-stale-replay-token-order.md` | Prevent a stale replay invocation from invalidating an already-running replay for the current session. | A late stale invocation exits before advancing the replay token; the current session still restores normally. |
 | 50 | done | `docs/plans/todo-50-comparable-model-tool-metrics.md` | Record mode-aware model/tool spans and include EventAgent usage so both architectures can be compared fairly. | Persistence/API tests report agent, call kind, tokens, TTFT, tool latency, fallback usage, and total turn wall time without double counting. |
 | 50.1 | done | `docs/plans/todo-50.1-metric-key-migration-dedup.md` | Deduplicate legacy metric keys before creating the idempotency index so existing experiment databases remain openable. | Migration keeps the earliest row for each duplicate non-null session/metric key, creates the unique index, and remains safe under concurrent initialization. |
-| 51 | pending | `docs/plans/todo-51-comparison-matrix-closeout.md` | Add repeatable comparison fixtures, update project documentation, and close the experimental implementation. | Both modes pass the same scenario matrix; full tests, JS syntax, app import, and documentation checks pass. |
+| 51 | in_progress | `docs/plans/todo-51-comparison-matrix-closeout.md` | Add repeatable comparison fixtures, update project documentation, and close the experimental implementation. | Both modes pass the same scenario matrix; full tests, JS syntax, app import, and documentation checks pass. |