# Todo 50 Comparable Model and Tool Metrics Plan

**Status:** in_progress

## Goal

Make `dual_agent` and `chat_agent_tools` directly comparable without double-counting model tokens, parallel latency, or replayed events.

## Metric Contract

- Keep the existing `/api/sessions/{id}/usage` endpoint and `calls` / `turns` / `session` shape.
- Extend the persisted call ledger with:
  - `agent`: `chat_agent`, `event_agent`, or `event_kernel`;
  - `call_kind`: `chat_completion`, `argument_fallback`, or `tool_execution`;
  - `event_id`, `event_name`, `used_fallback`, and `tool_latency_ms`;
  - deterministic `metric_key` for idempotent persistence.
- Derive `tool_invocation_mode` from the immutable session config instead of duplicating it per row.
- Add independent `turns.wall_time_ms`, measured once from the runtime monotonic clock.
- Token totals include only real model calls. Tool rows carry zero tokens.
- Turn/session wall time is calculated only from `turns.wall_time_ms`; model elapsed and concurrent tool latencies remain per-call diagnostics and are never summed as wall time.
- Persist only logical event results (`deduplicated_from is None`) so coalesced/replayed clones do not inflate tool or fallback counts.

## Runtime Scope

- ChatAgent records model TTFT from the first provider stream item while preserving visible-text TTFT in existing round events.
- EventAgent consumes the complete fallback stream so trailing usage is retained, and exposes one model-call metric per fallback attempt, including failed resolution attempts.
- Event kernel records handler/tool latency on each result.
- Direct mode persists ChatAgent model calls plus logical tool executions.
- Dual mode additionally persists EventAgent fallback model calls.

## Storage and UI Scope

- Migrate legacy SQLite databases with safe additive columns/indexes; legacy rows default to `chat_agent` / `chat_completion`.
- Extend call, turn, and session summaries with mode, call counts, fallback counts, tool counts, and independent wall time.
- Keep legacy `elapsed_ms` fields for compatibility; UI labels architecture/call kind and prefers `turn_wall_time_ms` for totals.

## Files

- Modify: `src/agent_lab/domain/messages.py`
- Modify: `src/agent_lab/application/session_store.py`
- Modify: `src/agent_lab/application/event_agent.py`
- Modify: `src/agent_lab/application/events/models.py`
- Modify: `src/agent_lab/application/events/kernel.py`
- Modify: `src/agent_lab/application/runtime.py`
- Modify: `src/agent_lab/infrastructure/sqlite_store.py`
- Modify: `src/agent_lab/presentation/static/app.js`
- Modify focused tests for event kernel/agent, runtime, SQLite, and WebSocket/UI.

## Requirement Execution Loop

### Plan

1. Add RED storage/API tests for typed call rows, legacy migration, mode derivation, idempotent metric keys, and wall-time aggregation.
2. Add RED EventAgent/kernel tests for trailing usage, first-provider-item TTFT, fallback attempts, and tool latency.
3. Add RED runtime parity tests proving direct and dual persist the same logical tool count while dual includes fallback model cost only when used.
4. Implement the smallest additive schema, metric capture, and UI formatting changes.
5. Run focused tests, full suite, JS syntax, migration stress, diff check, then specification and quality review.

## Verification

- `uv run pytest tests/test_event_kernel.py tests/test_event_agent.py tests/test_sqlite_store.py tests/test_debug_runtime.py tests/test_websocket_api.py -q`
- `node --check src/agent_lab/presentation/static/app.js`
- `uv run pytest`
- repeated legacy/concurrent SQLite migration test
- `git diff --check`