agent-lab/docs/plans/todo-50-comparable-model-tool-metrics.md

Todo 50 Comparable Model and Tool Metrics Plan

Status: in_progress

Goal

Make dual_agent and chat_agent_tools directly comparable without double-counting model tokens, parallel latency, or replayed events.

Metric Contract

• Keep the existing /api/sessions/{id}/usage endpoint and calls / turns / session shape.
• Extend the persisted call ledger with:
  • agent: chat_agent, event_agent, or event_kernel;
  • call_kind: chat_completion, argument_fallback, or tool_execution;
  • event_id, event_name, used_fallback, and tool_latency_ms;
  • deterministic metric_key for idempotent persistence.
• Derive tool_invocation_mode from the immutable session config instead of duplicating it per row.
• Add independent turns.wall_time_ms, measured once from the runtime monotonic clock.
• Token totals include only real model calls. Tool rows carry zero tokens.
• Turn/session wall time is calculated only from turns.wall_time_ms; model elapsed and concurrent tool latencies remain per-call diagnostics and are never summed as wall time.
• Persist only logical event results (deduplicated_from is None) so coalesced/replayed clones do not inflate tool or fallback counts.

Runtime Scope

• ChatAgent records model TTFT from the first provider stream item while preserving visible-text TTFT in existing round events.
• EventAgent consumes the complete fallback stream so trailing usage is retained, and exposes one model-call metric per fallback attempt, including failed resolution attempts.
• Event kernel records handler/tool latency on each result.
• Direct mode persists ChatAgent model calls plus logical tool executions.
• Dual mode additionally persists EventAgent fallback model calls.

Storage and UI Scope

• Migrate legacy SQLite databases with safe additive columns/indexes; legacy rows default to chat_agent / chat_completion.
• Extend call, turn, and session summaries with mode, call counts, fallback counts, tool counts, and independent wall time.
• Keep legacy elapsed_ms fields for compatibility; UI labels architecture/call kind and prefers turn_wall_time_ms for totals.

Files

• Modify: src/agent_lab/domain/messages.py
• Modify: src/agent_lab/application/session_store.py
• Modify: src/agent_lab/application/event_agent.py
• Modify: src/agent_lab/application/events/models.py
• Modify: src/agent_lab/application/events/kernel.py
• Modify: src/agent_lab/application/runtime.py
• Modify: src/agent_lab/infrastructure/sqlite_store.py
• Modify: src/agent_lab/presentation/static/app.js
• Modify focused tests for event kernel/agent, runtime, SQLite, and WebSocket/UI.

Requirement Execution Loop

Plan

1. Add RED storage/API tests for typed call rows, legacy migration, mode derivation, idempotent metric keys, and wall-time aggregation.
2. Add RED EventAgent/kernel tests for trailing usage, first-provider-item TTFT, fallback attempts, and tool latency.
3. Add RED runtime parity tests proving direct and dual persist the same logical tool count while dual includes fallback model cost only when used.
4. Implement the smallest additive schema, metric capture, and UI formatting changes.
5. Run focused tests, full suite, JS syntax, migration stress, diff check, then specification and quality review.

Verification

• uv run pytest tests/test_event_kernel.py tests/test_event_agent.py tests/test_sqlite_store.py tests/test_debug_runtime.py tests/test_websocket_api.py -q
• node --check src/agent_lab/presentation/static/app.js
• uv run pytest
• repeated legacy/concurrent SQLite migration test
• git diff --check