agent-lab/docs/plans/todo-12-round-stats.md

Todo 12 Structured Round Stats Plan

Status: done

Goal

Move round statistics to structured backend events for token counts, cached tokens, TTFT, elapsed time, and per-turn summaries.

Why

The original requirement calls for chat-window round statistics: token count, cache hits, TTFT, and total elapsed time. The MVP UI calculates TTFT/elapsed in the browser and only displays usage chunks. That is useful visually, but not enough for reproducible Agent debugging. Backend-emitted stats should become part of the WebSocket event stream.

Scope

This todo covers:

• Backend timing collection inside DebugRuntime.run().
• Structured round_stats WebSocket messages.
• Usage aggregation from streamed usage items.
• UI rendering from backend round_stats events.
• Tests for deterministic stats using an injectable clock.

This todo does not cover:

• Persistent run history.
• Multi-run comparison dashboards.
• Provider-side detailed cache diagnostics beyond available cached_tokens.
• Cost estimation.

Files

• Modify: src/agent_lab/application/runtime.py
• Modify/add application contracts or domain models only if needed.
• Modify: src/agent_lab/presentation/static/app.js
• Modify/add tests, preferably tests/test_debug_runtime.py and tests/test_websocket_api.py.
• Do not modify: docs/plans/todos.md.

Design

Add an injectable monotonic clock to DebugRuntime:

Clock = Callable[[], float]
DebugRuntime(chat_client, queues=None, registry=None, clock=time.perf_counter)

Track one round_index per ChatAgent call. For each round:

• started_at = clock();
• first message delta sets ttft_ms if unset;
• usage item updates prompt_tokens, completion_tokens, total_tokens, and cached_tokens;

• after the ChatAgent stream completes, emit:

  {
  "type": "round_stats",
  "round_index": 1,
  "ttft_ms": 123,
  "elapsed_ms": 456,
  "prompt_tokens": 10,
  "completion_tokens": 20,
  "total_tokens": 30,
  "cached_tokens": 5,
  "had_event": true
  }
  

Keep existing usage messages for compatibility, but the UI should prefer round_stats for the stats panel. If no token arrives, ttft_ms can be null.

Steps

1. Add failing runtime tests for:
   • round_stats is emitted after a model turn;
   • TTFT and elapsed are derived from the injected clock;
   • usage totals are included;
   • multi-turn event handoff emits one stats object per ChatAgent call.
2. Refactor DebugRuntime to collect and emit stats through the existing output queue.
3. Update app.js to update stats from round_stats messages and keep current browser timing only as a fallback before stats arrives.
4. Add or update static JS tests/assertions for round_stats handling.

5. Run:

   uv run pytest tests/test_debug_runtime.py tests/test_websocket_api.py
   uv run pytest
   

6. If successful, commit implementation files only with:

   git commit -m "feat: emit structured round stats"
   

Verification

Result:

• uv run pytest tests/test_debug_runtime.py tests/test_websocket_api.py passed: 23 tests, 1 existing Starlette deprecation warning.
• uv run pytest passed: 28 tests, 1 existing Starlette deprecation warning.

Evaluation

• Backend emits deterministic round_stats messages with round index, TTFT, elapsed time, token totals, cached tokens, and event presence.
• UI handles backend round_stats and keeps browser-side timing as a fallback.
• The original MVP requirement for chat-window round statistics is covered.