docs: document invocation comparison experiment

Problem:
- README did not expose the implemented shared-kernel comparison workflow, namespaced events, operational surfaces, or fair usage contract.
- The production-events design still used legacy short event identifiers and lacked implementation/closeout links.

Risk:
- Documentation only; default adapters remain in-memory/mock and live provider or service integrations are not verified here.
- scripts/ws_smoke.py remains a connectivity smoke, not a dual-mode benchmark.

README.md

@@ -1,68 +1,202 @@
 # Agent Lab
 
-Agent Lab is a FastAPI debug console for chat agents and event agents. It uses
-an OpenAI Chat Completions compatible streaming endpoint, a WebSocket debug
-route, and a direct async runtime/event flow for local investigation.
+Agent Lab is a FastAPI debug console for comparing ChatAgent capability-invocation
+protocols on one shared event runtime. The implementation uses a generic event
+kernel plus a flat registry of built-in plugins; invocation mode changes how the
+model selects capabilities, not how accepted events are normalized, validated,
+scheduled, executed, persisted, or reported.
 
-## Setup
+The architecture source of truth is the
+[production events design](docs/plans/2026-07-12-chat-event-agent-production-events-design.md).
+The repeatable experiment and closeout scope is tracked by
+[Todo 51](docs/plans/todo-51-comparison-matrix-closeout.md).
 
-Install dependencies with uv:
+## Architecture and Built-ins
 
-```bash
-uv sync
-```
+The generic kernel owns event lookup, deterministic resolution, validation,
+bounded fallback, deduplication, conflict-aware concurrency, timeouts, terminal
+barriers, normalized results, result-policy aggregation, audit events, and
+metrics. Runtime and the kernel do not branch on a built-in event name.
+
+The four production-shaped plugins are flat, namespaced definitions:
+
+| Event | Result policy | Current default behavior |
+| --- | --- | --- |
+| `session.terminate` | `terminate` | Records an in-memory terminated result and ends the debug session on success. |
+| `device.volume.adjust` | `silent_success` | Validates and records an in-memory volume action; success adds no second answer. |
+| `calendar.schedule.create` | `template_follow_up` | Records an in-memory schedule and emits a deterministic confirmation. |
+| `knowledge.web.search` | `llm_follow_up` | Returns deterministic mock sources, then asks ChatAgent for one grounded follow-up. |
+
+The default registry also retains the lab-only `handoff_note`, `mock_search`, and
+`mock_ticket` tools. `GET /api/tools` is the authoritative runtime catalog.
+
+## Invocation Modes
+
+Set `tool_invocation_mode` per request:
+
+| Mode | Selection protocol | Argument source | Mode-specific cost or dependency |
+| --- | --- | --- | --- |
+| `dual_agent` | ChatAgent receives a compact catalog and emits hidden text events after visible content. | Shared deterministic resolvers run first; EventAgent may make a bounded fallback model call for incomplete supported events. | Depends on marker adherence and adds fallback calls when deterministic resolution is incomplete. |
+| `chat_agent_tools` | ChatAgent receives enabled provider tool schemas and emits native tool calls. | ChatAgent proposes arguments; the shared kernel still normalizes, validates, authorizes, deduplicates, schedules, and executes them. | Full schemas and provider tool transcript remain in ChatAgent context and usage; EventAgent model/prompt settings are inactive for provider-resolved calls. |
+
+Both modes share the same enabled definitions, JSON schemas, handlers/adapters,
+event budget, batch concurrency, timeout, conflict and terminal policies,
+normalized results, and user-visible success/failure contract. Provider-native
+assistant/tool transcript details are intentionally allowed to differ.
+
+### Web search always has two answers
+
+For `knowledge.web.search`, both modes implement the same visible sequence:
+
+1. ChatAgent emits a useful or qualified first answer.
+2. The shared pipeline executes the search and emits its tool result.
+3. ChatAgent makes exactly one follow-up model call and emits a distinct second answer.
+
+The first answer must not wait for the search adapter, and the second answer must
+use the compact returned evidence. This is a shared behavior contract, not a
+provider transcript normalization trick.
+
+## Current Adapter Limits
 
-Create a local environment file:
+The default application wiring is safe for local experiments, but it is not
+production integration:
+
+- session termination is an in-memory result, not a real dialogue-service command;
+- volume adjustment does not control a device or publish MQTT/device actions;
+- schedule creation does not write to a real calendar or reminder service;
+- web search returns deterministic `example.invalid` mock sources and does not access the internet.
+
+Inject implementations of `SessionTerminationPort`, `DeviceVolumePort`,
+`CalendarSchedulePort`, and `WebSearchPort` when testing service-owned adapters.
+Real device, calendar, session, and internet integration belongs outside the
+default Agent Lab wiring.
+
+## Configure and Run
+
+Install dependencies and create a local environment file:
 
 ```bash
+uv sync
 cp .env.example .env
 ```
 
-Set these values in `.env` before running real LLM calls:
-
-- `AGENT_LAB_OPENAI_API_KEY`: required API key for the compatible provider.
-- `AGENT_LAB_OPENAI_BASE_URL`: provider `/v1` root, for example `https://api.openai.com/v1`.
-- `AGENT_LAB_OPENAI_DEFAULT_MODEL`: backend default model when a request leaves model blank.
-- `AGENT_LAB_OPENAI_INCLUDE_USAGE`: set `false` if the compatible provider rejects `stream_options.include_usage`.
-- `AGENT_LAB_REQUEST_TIMEOUT_SECONDS`: HTTP timeout for streamed model calls.
-
-Do not commit real secrets. Keep `.env` local.
+Supported settings use the `AGENT_LAB_` prefix:
 
-## Run Locally
+- `AGENT_LAB_OPENAI_API_KEY`: API key for the compatible provider;
+- `AGENT_LAB_OPENAI_BASE_URL`: provider `/v1` root;
+- `AGENT_LAB_OPENAI_DEFAULT_MODEL`: model used when a request leaves `model` blank;
+- `AGENT_LAB_OPENAI_INCLUDE_USAGE`: disable if the provider rejects streamed usage options;
+- `AGENT_LAB_REQUEST_TIMEOUT_SECONDS`: provider HTTP timeout;
+- `AGENT_LAB_DATABASE_PATH`: SQLite session/audit/usage database path (defaults to `.agent_lab.sqlite3`).
 
-Start the FastAPI app:
+Do not commit secrets. Start the app with:
 
 ```bash
 uv run uvicorn agent_lab.main:app --reload
 ```
 
-Open the debug console:
+Open `http://127.0.0.1:8000`. The UI can select either invocation mode, enable
+tools/events, configure ChatAgent and applicable EventAgent settings, set event
+loop/concurrency/batch limits, create or restore sessions, replay messages and
+audit data, and inspect per-call, per-turn, and per-session usage.
 
-```text
-http://127.0.0.1:8000
+## WebSocket
+
+Connect to `ws://127.0.0.1:8000/ws/debug`. The smallest valid initial payload is:
+
+```json
+{
+  "user_message": "Hello",
+  "chat_agent": {}
+}
+```
+
+This uses the default `dual_agent` mode and default EventAgent configuration. A
+minimal built-in event request can make the comparison controls explicit:
+
+```json
+{
+  "user_message": "Set the device volume to 30",
+  "chat_agent": {},
+  "event_agent": {
+    "enabled_tools": ["device.volume.adjust"]
+  },
+  "tool_invocation_mode": "dual_agent"
+}
 ```
 
-The app exposes `GET /health`, `GET /`, and `WebSocket /ws/debug`.
+Change only `tool_invocation_mode` to `chat_agent_tools` for the other protocol
+while keeping model parameters, enabled events, event limits, input, and adapter
+wiring equal. After `session_started`, the same socket accepts later turns as
+`{"type":"user_message","content":"..."}` until a terminal `done` or `error`.
 
-## WebSocket Smoke Test
+### `ws_smoke.py` is not a dual-mode benchmark
 
-With the server running and `.env` configured, send one debug request:
+With the server running, `scripts/ws_smoke.py` sends one convenience request and
+prints streamed JSON:
 
 ```bash
-uv run python scripts/ws_smoke.py \
-  --message "Debug this agent handoff path."
+uv run python scripts/ws_smoke.py --message "Debug this agent handoff path."
 ```
 
-The script connects to `ws://127.0.0.1:8000/ws/debug`, sends a valid
-`DebugRunRequest`, and prints each server message as compact JSON. Use
-`--model` to override the backend default model, `--disable-handoff-note` to
-run without the default `handoff_note` tool, or `--url` to target a different
-server.
+It defaults to `dual_agent`, exercises the `handoff_note` path, has no paired
+mode runner or fairness controls, and does not collect the Todo 51 scenario
+matrix. Use it only for WebSocket connectivity and payload smoke testing.
+
+## HTTP API
+
+| Method and path | Purpose |
+| --- | --- |
+| `GET /health` | Process health. |
+| `GET /` | Static debug UI. |
+| `GET /api/tools` | Current flat registry catalog and schemas. |
+| `POST /api/sessions` | Create a persisted session with optional `title` and `config`. |
+| `GET /api/sessions` | List persisted sessions. |
+| `GET /api/sessions/{session_id}` | Read session metadata and immutable comparison config. |
+| `GET /api/sessions/{session_id}/messages` | Replay persisted visible/provider messages. |
+| `GET /api/sessions/{session_id}/audit` | Read ordered runtime audit events. |
+| `GET /api/sessions/{session_id}/usage` | Read comparable call, turn, and session metrics. |
+
+## `/usage` Fair-Comparison Contract
+
+`GET /api/sessions/{session_id}/usage` returns `calls`, `turns`, and `session`.
+Interpret it with these controls:
+
+- compare runs with the same model parameters, enabled event set, user input,
+  output budget, registry/adapters, event budget, concurrency, and timeouts;
+- token totals include real ChatAgent and EventAgent model calls only; tool
+  execution rows have zero tokens;
+- direct-mode provider tool schema and assistant/tool protocol tokens remain
+  counted rather than being normalized away;
+- EventAgent argument-fallback calls are separate `argument_fallback` rows and
+  contribute their model tokens;
+- `turn_wall_time_ms` is the turn/session wall-clock measure. Do not add model
+  `elapsed_ms` or overlapping `tool_latency_ms` spans to estimate total latency;
+- only logical tool executions count: deduplicated/replayed clones do not inflate
+  `tool_count` or fallback count;
+- call-level `ttft_ms` measures the first provider stream item. Visible-text TTFT
+  is a separate runtime/audit observation and must not be conflated with it.
+
+## Verification
+
+Run the closeout matrix and focused runtime checks:
 
-## Tests
+```bash
+uv run pytest tests/test_tool_invocation_comparison.py -q
+uv run pytest tests/test_debug_runtime.py tests/test_builtin_event_plugins.py tests/test_sqlite_store.py tests/test_websocket_api.py -q
+```
 
-Run the full test suite:
+Run the repository gates:
 
 ```bash
 uv run pytest
+node --check src/agent_lab/presentation/static/app.js
+uv run python -c "from agent_lab.main import app; print(app.title)"
+git diff --check
 ```
+
+These tests establish repeatable application semantics with scripted model
+clients and injected/default mock ports. They do not prove live-provider content
+plus-tool streaming behavior, real network search quality, real device/calendar
+side effects, production credentials, or production latency. Run provider and
+service integration smokes separately before drawing deployment conclusions.

docs/plans/2026-07-12-chat-event-agent-production-events-design.md

@@ -1,9 +1,17 @@
 # ChatAgent / EventAgent Production Events Design
 
-**Status:** approved design
+**Status:** implemented in Agent Lab by Todos 45-50; Todo 51 closeout in progress
 
 **Date:** 2026-07-12
 
+**Implementation links:** [Todo 45 generic event kernel](todo-45-generic-event-kernel.md),
+[Todo 46 invocation modes](todo-46-configurable-tool-invocation-mode.md),
+[Todo 47 built-in plugins](todo-47-built-in-event-plugins.md),
+[Todo 48 parallel batches](todo-48-parallel-event-batches.md),
+[Todo 49 UI/session restore](todo-49-tool-mode-ui-session-restore.md),
+[Todo 50 comparable metrics](todo-50-comparable-model-tool-metrics.md), and
+[Todo 51 comparison closeout](todo-51-comparison-matrix-closeout.md).
+
 ## 1. Goal
 
 Evaluate two configurable capability-invocation architectures for the service voice-reply path, primarily targeting a low-latency conversational toy:
@@ -49,10 +57,10 @@ Initial mapping:
 
 | Event | Result policy |
 | --- | --- |
-| `terminate_session` | `terminate` |
-| `adjust_device_volume` | `silent_success` |
-| `add_schedule` | `template_follow_up` |
-| `web_search` | `llm_follow_up` |
+| `session.terminate` | `terminate` |
+| `device.volume.adjust` | `silent_success` |
+| `calendar.schedule.create` | `template_follow_up` |
+| `knowledge.web.search` | `llm_follow_up` |
 
 ### 2.1 Experimental consistency contract
 
@@ -62,7 +70,7 @@ The comparison must not become two separate implementations. Both modes use:
 - the same ChatAgent model, persona prompt, user-visible conversational history, sampling parameters, and output budget, except for the minimum mode-specific event/tool instruction;
 - the same normalized internal `EventBatch` and event IDs;
 - the same deterministic resolvers, validators, optional low-cost LLM fallback, handlers, timeouts, retries, idempotency keys, concurrency rules, and terminal barriers;
-- the same result policies and user-visible behavior, including the two-answer `web_search` flow;
+- the same result policies and user-visible behavior, including the two-answer `knowledge.web.search` flow;
 - the same audit event names and metric definitions.
 
 Only these variables change:
@@ -149,7 +157,7 @@ The flat plugin registry should evolve only when real additions reveal repeated
 
 1. The user expresses a clear intention to leave or end the voice conversation.
 2. ChatAgent immediately streams a short, natural farewell.
-3. ChatAgent requests `terminate_session` through the configured text-event or native tool-call protocol.
+3. ChatAgent requests `session.terminate` through the configured text-event or native tool-call protocol.
 4. EventAgent executes a dialogue-session termination command.
 5. On success, the runtime marks the session terminal and suppresses another ChatAgent round.
 6. On failure, the runtime records the failure and may emit one concise corrective message instead of pretending that the session ended.
@@ -160,7 +168,7 @@ This event should terminate the dialogue through the dialogue/session boundary.
 
 1. The user asks for an absolute volume, relative change, mute, or unmute.
 2. ChatAgent immediately acknowledges the requested action naturally.
-3. ChatAgent requests `adjust_device_volume` through the configured protocol.
+3. ChatAgent requests `device.volume.adjust` through the configured protocol.
 4. EventAgent resolves normalized arguments such as:
    - `mode`: `absolute`, `relative`, `mute`, or `unmute`;
    - `value`: an absolute target when present;
@@ -173,7 +181,7 @@ Validation must clamp or reject values outside the supported range and must dist
 ### 3.3 Add schedule
 
 1. The user asks the toy to remember a future activity, reminder, or calendar item.
-2. ChatAgent immediately acknowledges the request naturally and requests `add_schedule` through the configured protocol.
+2. ChatAgent immediately acknowledges the request naturally and requests `calendar.schedule.create` through the configured protocol.
 3. EventAgent first attempts deterministic extraction and normalization of:
    - title or activity;
    - local date and time;
@@ -189,7 +197,7 @@ Schedule execution must use an idempotency key so retransmission or duplicate ev
 ### 3.4 Web search with two answers
 
 1. ChatAgent immediately gives a useful preliminary response in a natural tone.
-2. When freshness or external facts matter, ChatAgent mentions that it will check and requests `web_search` through the configured protocol after the visible reply where the provider supports content plus tool calls.
+2. When freshness or external facts matter, ChatAgent mentions that it will check and requests `knowledge.web.search` through the configured protocol after the visible reply where the provider supports content plus tool calls.
 3. EventAgent derives a focused search query from the current user request and relevant conversation history.
 4. The search adapter returns structured results, including query, answer evidence, source title, source location, and retrieval time where available.
 5. The runtime sends a compact search-result summary back to ChatAgent.
@@ -215,8 +223,8 @@ Examples that may run concurrently:
 Conflict policy:
 
 - duplicate event names are deduplicated only when their normalized arguments are equivalent;
-- read-only events such as `web_search` may run alongside side-effecting events;
-- `terminate_session` is dispatched after other accepted events have reached a terminal result, with a short overall deadline, so ending the dialogue does not cancel an already acknowledged schedule or device action;
+- read-only events such as `knowledge.web.search` may run alongside side-effecting events;
+- `session.terminate` is dispatched after other accepted events have reached a terminal result, with a short overall deadline, so ending the dialogue does not cancel an already acknowledged schedule or device action;
 - an event failure does not cancel unrelated events;
 - follow-up outputs are aggregated by policy: deterministic confirmations first, then at most one ChatAgent LLM follow-up for all results that require natural synthesis.
 
@@ -274,7 +282,7 @@ EventAgent should receive:
 - the smallest relevant slice of user/assistant conversation history;
 - EventAgent-specific instructions and runtime metadata needed for execution.
 
-For `web_search`, only the compact result summary enters the second ChatAgent call. Raw provider payloads stay in audit/debug data.
+For `knowledge.web.search`, only the compact result summary enters the second ChatAgent call. Raw provider payloads stay in audit/debug data.
 
 For deterministic resolution, no EventAgent LLM request is made. When fallback is needed, EventAgent should use a lower-cost model and a small token limit because its output is structured arguments rather than conversational text.
 
@@ -319,10 +327,10 @@ Each repository plan must load that repository's `AGENTS.md` and maintain its ow
 - Both modes are prompted to emit the natural first response before event execution; if a native tool provider suppresses visible content until after tool execution, that is recorded as a direct-mode behavior/consistency failure rather than hidden by the harness.
 - Hidden event blocks never appear in user-visible deltas.
 - Deterministic argument resolution bypasses the EventAgent LLM, while incomplete or ambiguous cases use the configured fallback.
-- `terminate_session` yields farewell -> accepted sibling events finish or time out -> successful termination -> no second ChatAgent call.
-- `adjust_device_volume` yields acknowledgement -> action result, with no second answer on success and a corrective answer on failure.
-- `add_schedule` yields acknowledgement -> normalized execution -> deterministic confirmation, without a second LLM call.
-- `web_search` yields first answer -> search execution -> distinct second answer grounded in the returned result.
+- `session.terminate` yields farewell -> accepted sibling events finish or time out -> successful termination -> no second ChatAgent call.
+- `device.volume.adjust` yields acknowledgement -> action result, with no second answer on success and a corrective answer on failure.
+- `calendar.schedule.create` yields acknowledgement -> normalized execution -> deterministic confirmation, without a second LLM call.
+- `knowledge.web.search` yields first answer -> search execution -> distinct second answer grounded in the returned result.
 - Multiple independent events execute concurrently, preserve input order in reported results, and isolate failures.
 - A mixed batch produces at most one extra ChatAgent LLM call even when several events require natural follow-up.
 - Search failure still yields a natural, non-blocking second message.