todos.md 28 KB

Agent Lab MVP Todos

Workflow

  • Main session owns this file and evaluates progress after each todo.
  • Work on one todo at a time: write its plan, execute it, verify it, then update this file.
  • Do not plan all implementation details upfront. Later todos may change after each evaluation.
  • Detailed plans live beside this file as docs/plans/todo-N-*.md.
  • Status values: pending, in_progress, done, blocked.

Current State

  • Branch: codex/agent-lab-mvp.
  • MVP committed at 6babafa with FastAPI, WebSocket, static UI, OpenAI-compatible streaming, EventAgent handoff, provider-compatibility guards, and tests.
  • Post-MVP work now evolves toward the original queue/event-driven debugging-console requirements.

Coverage Review

  • OpenAI Chat Completions compatible streaming and provider payload guards are covered by todos 2, 5, and 7.
  • Queue/event-driven ChatAgent and EventAgent handoff is covered by todo 9.
  • EventAgent tool loading and UI management is covered by todo 10.
  • Prompt/pre-message editing, parameter controls, and reusable prompt sets are covered by todos 3 and 11.
  • Chat-window token, cached-token, TTFT, elapsed-time, and per-round stats are covered by todo 12.
  • Review follow-up gaps in real queue boundaries, API validation, packaging, and tool error isolation are covered by todos 13-16.

Todos

ID Status Plan Goal Verification
1 done docs/plans/todo-1-project-foundation.md Establish uv/FastAPI project skeleton, DDD package layout, and core failing tests for parser, event agent, and runtime routing. uv run pytest fails only because implementation is missing, not because tests or imports are malformed.
2 done docs/plans/todo-2-domain-runtime.md Implement domain models, OpenAI Chat Completions compatible stream parsing, EventAgent tools, and the direct async runtime loop. Core tests pass with mocked ChatAgent client.
3 done docs/plans/todo-3-websocket-api.md Add FastAPI config, HTTP routes, WebSocket protocol, static debug page, and assistant transcript persistence for multi-turn LLM calls. WebSocket contract tests pass; app imports successfully.
4 done docs/plans/todo-4-real-llm-smoke.md Document env-based real LLM run flow, add a lightweight manual smoke helper, and make sure settings names are clear. App starts with env-based config; docs show exact uv commands and required env vars.
5 done docs/plans/todo-5-tool-call-history.md Harden Chat Completions history so assistant tool calls and tool replies are sent in provider-compatible order. Tests prove the next model call includes assistant tool_calls before tool replies.
6 done docs/plans/todo-6-cleanup-closeout.md Clean temporary/generated files, update contributor docs, and run final verification. Git status contains only intended source/docs files; uv run pytest passes.
7 done docs/plans/todo-7-provider-compatibility.md Fix review findings for real Chat Completions compatible providers: default model fallback, role-aware message serialization, and configurable usage streaming. Tests prove provider payload shape and full suite passes.
8 done docs/plans/todo-8-tool-pre-message-guard.md Prevent manually configured pre-messages from producing invalid provider tool payloads without tool_call_id. Tests prove invalid tool pre-messages are rejected and UI no longer exposes tool role.
9 done docs/plans/todo-9-explicit-runtime-queues.md Refactor runtime internals to explicit input, output, and event queues while preserving the WebSocket stream contract. Tests prove ChatAgent consumes input queue, EventAgent consumes event queue, and output queue drives streamed responses.
10 done docs/plans/todo-10-tool-registry-management.md Replace the single hardcoded EventAgent tool checkbox with a backend tool registry and UI-managed loaded tools. Tests prove available tools are listed by API and selected tools control EventAgent behavior.
11 done docs/plans/todo-11-prompt-workspace.md Improve prompt/pre-message editing with browser-side persistence and reusable prompt sets. UI tests or focused JS tests prove prompts persist and can be restored.
12 done docs/plans/todo-12-round-stats.md Move round statistics to structured backend events for token counts, cached tokens, TTFT, elapsed time, and per-turn summaries. uv run pytest tests/test_debug_runtime.py tests/test_websocket_api.py; uv run pytest.
13 done docs/plans/todo-13-real-queue-websocket-loop.md Make runtime queues real producer/consumer boundaries and let WebSocket run separate upstream/downstream tasks. uv run pytest tests/test_debug_runtime.py tests/test_websocket_api.py; uv run pytest.
14 done docs/plans/todo-14-message-role-validation.md Reject provider-invalid API message roles and malformed tool replies before network calls. uv run pytest tests/test_websocket_api.py tests/test_event_agent.py; uv run pytest.
15 done docs/plans/todo-15-static-package-data.md Include static UI assets in installed package builds. uv run pytest tests/test_packaging.py; uv run pytest; uv build; wheel inspection.
16 done docs/plans/todo-16-tool-error-isolation.md Convert EventAgent tool handler exceptions into structured tool-result errors instead of aborting the session. uv run pytest tests/test_event_agent.py tests/test_debug_runtime.py; uv run pytest.
17 done docs/plans/todo-17-agent-config-audit-tools.md Align ChatAgent/EventAgent config, add extra_body defaults, default EventAgent to one event round, add mock tools, audit events, and backend logging. uv run pytest passes (42 passed, one existing Starlette deprecation warning).
18 done docs/plans/todo-18-prompt-modal-tool-ui.md Move prompt workspace into a modal and improve visible tool-management UI. uv run pytest passes (46 passed), plus local browser check confirms modal and tool UI render.
19 done docs/plans/todo-19-latency-oriented-polish.md Do one optimization pass focused on reducing perceived reply wait time and cleaning frontend/backend rough edges. uv run pytest passes (49 passed), plus local browser check confirms tools are visible in the first viewport.
20 done docs/plans/todo-20-agent-config-modals.md Split ChatAgent and EventAgent settings into separate configuration modals so the sidebar is no longer crowded. uv run pytest passes.
21 done docs/plans/todo-21-event-agent-context-summary.md Make EventAgent config part of parameter generation context and enqueue an aggregated tool-result summary for the next ChatAgent round. uv run pytest passes (50 passed, one existing Starlette deprecation warning).
22 done docs/plans/todo-22-text-event-protocol.md Replace ChatAgent tool calls with streamed text event parsing, inject generated event descriptions as a system message, and harden Agent config buttons. uv run pytest passes (50 passed, one existing Starlette deprecation warning), plus local browser click check.
23 done docs/plans/todo-23-agent-prompt-and-event-llm.md Move Prompt Config into ChatAgent config, show available-event prompt by default, and make EventAgent use LLM tool calls to generate tool arguments. uv run pytest passes (51 passed, one existing Starlette deprecation warning); changed Python modules compile; local server returned page/static assets/tools API.
24 done docs/plans/todo-24-chat-agent-prompt-list.md Put all ChatAgent prompts, including generated EventAgent event rules, into one ordered prompt list with add/delete/drag controls. uv run pytest passes (51 passed, one existing Starlette deprecation warning), plus local browser prompt-list check.
25 done docs/plans/todo-25-prompt-list-message-types.md Make the ChatAgent prompt list visually compact and show the message type for each prompt item. uv run pytest passes (51 passed, one existing Starlette deprecation warning); local service returned /health and versioned JS asset.
26 done docs/plans/todo-26-workspace-snapshot-save.md Move save/load/delete out of ChatAgent config and save one workspace snapshot covering prompts, Agent config, and selected tools. uv run pytest tests/test_websocket_api.py -q passes (27 passed, one existing Starlette deprecation warning).
27 done docs/plans/todo-27-event-agent-tool-arguments.md Make EventAgent tool argument generation robust for compatible providers that do not emit the expected tool-call finish reason or ignore tool calls. uv run pytest passes (53 passed, one existing Starlette deprecation warning).
28 done docs/plans/todo-28-session-turns.md Split WebSocket session lifetime from user turns so one conversation can run multiple user messages while each turn has its own event budget. uv run pytest passes (56 passed, one existing Starlette deprecation warning).
29 done docs/plans/todo-29-rich-audit-log.md Add audit entries for ChatAgent/EventAgent model parameters, prompts/messages, results, and usage without logging secrets. uv run pytest passes (57 passed, one existing Starlette deprecation warning).
30 done docs/plans/todo-30-sqlite-session-store.md Add a SQLite-backed store and REST APIs for sessions, chat messages, audit replay data, and usage summaries. uv run pytest tests/test_sqlite_store.py tests/test_websocket_api.py -q passes (36 passed, one existing Starlette deprecation warning); uv run pytest passes (66 passed, one existing warning).
31 done docs/plans/todo-31-runtime-persistence.md Persist runtime Agent config snapshots, user/assistant messages, audit entries, and per-call usage during WebSocket turns. uv run pytest tests/test_sqlite_store.py tests/test_debug_runtime.py tests/test_websocket_api.py -q passes (55 passed, one existing warning); uv run pytest passes (66 passed, one existing warning).
32 done docs/plans/todo-32-session-ui.md Add new-session/session-list UI, separate audit replay panel, and single-call/turn/session usage displays. node --check src/agent_lab/presentation/static/app.js; uv run pytest tests/test_websocket_api.py -q passes (34 passed, one existing warning); uv run pytest passes (66 passed, one existing warning).
33 done docs/plans/todo-33-audit-replay-stream-ui.md Redesign Audit Replay as a clickable event stream, keep audit events out of chat, and fix fixed/scrolling layout boundaries. node --check src/agent_lab/presentation/static/app.js; uv run pytest tests/test_websocket_api.py -q passes (36 passed, one existing warning); uv run pytest passes (68 passed, one existing warning); git diff --check passes.
34 done docs/plans/todo-34-event-agent-history-boundary.md Restrict EventAgent context to conversational user/assistant messages instead of the full ChatAgent context list. uv run pytest tests/test_debug_runtime.py -q passes (21 passed); node --check src/agent_lab/presentation/static/app.js; uv run pytest passes (70 passed, one existing warning); git diff --check passes.
35 done docs/plans/todo-35-audit-modal-raw-llm-events.md Make Audit Replay a modal, fix round event metadata after EventAgent follow-up rounds, and expose raw LLM response chunks. node --check src/agent_lab/presentation/static/app.js; uv run pytest tests/test_debug_runtime.py tests/test_event_agent.py tests/test_websocket_api.py -q passes (63 passed, one existing warning); uv run pytest passes (71 passed, one existing warning); git diff --check passes.
36 done docs/plans/todo-36-audit-detail-compact-ui.md Make Audit Detail modal metadata and event lists compact, and version CSS to avoid stale oversized styling. node --check src/agent_lab/presentation/static/app.js; uv run pytest tests/test_websocket_api.py -q passes (36 passed, one existing warning); uv run pytest passes (71 passed, one existing warning); git diff --check passes.
37 done docs/plans/todo-37-audit-detail-flow-layout.md Fully replace Audit Detail grid/card layout with compact flow layout so metadata cannot stretch vertically. node --check src/agent_lab/presentation/static/app.js; uv run pytest tests/test_websocket_api.py -q passes (36 passed, one existing warning); uv run pytest passes (71 passed, one existing warning); git diff --check passes.
38 done docs/plans/todo-38-audit-round-event-semantics.md Make events_enabled mean current-round enabled events and move full configured event list to configured_events. node --check src/agent_lab/presentation/static/app.js; uv run pytest tests/test_debug_runtime.py tests/test_websocket_api.py -q passes (58 passed, one existing warning); uv run pytest passes (71 passed, one existing warning); git diff --check passes.
39 done docs/plans/todo-39-audit-turn-relative-time.md Add per-audit-event elapsed time relative to the user message that started the current turn. uv run pytest passes (72 passed, one existing Starlette deprecation warning); node --check src/agent_lab/presentation/static/app.js; git diff --check passes.
40 done docs/plans/todo-40-chat-message-stream-audit.md Add audit events before ChatAgent visible message streaming starts and after it ends. uv run pytest passes (73 passed, one existing Starlette deprecation warning); node --check src/agent_lab/presentation/static/app.js; git diff --check passes.
41 done docs/plans/todo-41-session-button-feedback.md Add explicit Sessions-panel feedback for New, Refresh, and Load clicks. uv run pytest passes (73 passed, one existing Starlette deprecation warning); node --check src/agent_lab/presentation/static/app.js; git diff --check passes.
42 done docs/plans/todo-42-session-switch-idle-socket.md Allow switching sessions after a turn completes while the reusable WebSocket is still open. uv run pytest tests/test_websocket_api.py -q passes (37 passed, one existing Starlette deprecation warning); uv run pytest passes (74 passed, one existing warning); node --check src/agent_lab/presentation/static/app.js; git diff --check passes.
43 done docs/plans/todo-43-typed-provider-tool-calls.md Preserve native provider tool calls as a distinct stream event without changing the existing text-event path. uv run pytest tests/test_openai_stream_parser.py -q; focused typed-event regressions; full suite passes (82 passed, one existing warning).
43.1 done docs/plans/todo-43.1-typed-event-consumer-regression.md Repair Runtime and EventAgent consumers after typed stream kinds exposed legacy event assumptions. Runtime consumes text_event, EventAgent consumes provider_tool_call; focused and full tests pass.
43.2 done docs/plans/todo-43.2-remove-legacy-event-consumption.md Remove legacy event consumption from production paths and migrate remaining fixtures to typed stream kinds. Wrong-source negative tests and real client [DONE] integration pass; full suite passes (82 passed).
44 done docs/plans/todo-44-provider-tool-transcript.md Add provider-compatible assistant tool-call and tool-result message serialization plus explicit tool-choice behavior. Focused transcript tests pass (97 passed); full suite passes (108 passed, one existing warning).
44.1 done docs/plans/todo-44.1-tool-transcript-hardening.md Harden transcript identity, reserved payload fields, EventAgent history projection, and public pre-message boundaries. Duplicate IDs, invalid choices, reserved overrides, projected history, and pre-message boundaries are covered; full suite passes.
45 done docs/plans/todo-45-generic-event-kernel.md Build a generic event kernel with definition metadata, deterministic-first resolution, policies, execution ports, and result aggregation. Generic kernel, EventAgent compatibility, and event-name decoupling pass final gates; full suite passes (203 passed, one existing warning).
45.1 done docs/plans/todo-45.1-event-kernel-boundary-hardening.md Harden JSON Schema validation, normalized failures, fallback raw arguments, async handlers, and existing EventAgent concurrency. JSON Schema, normalized failure, strict argument, async handler, and concurrency regressions pass.
45.2 done docs/plans/todo-45.2-event-fallback-semantics.md Restrict fallback to incomplete resolution, enforce JSON-type-sensitive raw consistency, and distinguish definition failures. Fallback is limited to missing-only/incomplete-valid input; complete-invalid and provider-resolved input never call LLM fallback.
45.3 done docs/plans/todo-45.3-sync-resolution-completeness.md Make synchronous kernel execution reject explicitly incomplete deterministic resolution instead of invoking the handler. Sync structured incomplete input returns INVALID_ARGUMENTS without handler execution.
45.4 done docs/plans/todo-45.4-event-kernel-quality-boundaries.md Close reviewed configuration, composed-schema fallback, mutation isolation, registry encapsulation, and payload-serialization gaps. EventAgent config, mutation isolation, registry encapsulation, and strict JSON payload failures are covered.
45.5 done docs/plans/todo-45.5-composed-schema-and-copy-failures.md Prevent unrelated composed-schema branches from triggering fallback and normalize argument snapshot failures without exposing validator internals. Branch-aware composed validation and immutable validation summaries pass direct reproductions.
45.6 done docs/plans/todo-45.6-mixed-validation-and-handler-snapshot.md Require every validation failure to be missing-only before fallback and contain arbitrary handler payload serialization exceptions. Mixed validation skips fallback; arbitrary handler snapshot exceptions become stable HANDLER_ERROR.
45.7 done docs/plans/todo-45.7-structured-incomplete-mixed-validation.md Prevent an explicit structured complete=False flag from bypassing substantive validation failures. Structured incomplete plus substantive errors skips fallback; bounded final gate APPROVED.
46 done docs/plans/todo-46-configurable-tool-invocation-mode.md Add dual_agent and chat_agent_tools runtime modes on the shared event kernel. Both modes share registry/kernel/results while preserving mode-specific protocol and valid transcripts; full suite passes (217 passed, one existing warning).
46.1 done docs/plans/todo-46.1-mode-parity-and-duplicate-call-safety.md Reject repeated provider call IDs before side effects, align direct multi-call concurrency with dual mode, and enforce the same event budget in both modes. Duplicate IDs are rejected before side effects; direct batches overlap with stable order; dual/direct budgets are symmetric; final gates pass.
47 done docs/plans/todo-47-built-in-event-plugins.md Add terminate-session, device-volume, schedule, and web-search as built-in plugins on the generic event kernel. Four namespaced plugins use injectable ports/default adapters with no generic name branches; full suite passes (313 passed, one existing warning).
47.1 done docs/plans/todo-47.1-plugin-normalization-and-validation.md Share plugin normalization across invocation modes and harden negation, numeric, date/timezone, query, and adapter idempotency boundaries. Direct/dual normalization matches; strict date/timezone/query validation and canonical adapter keys pass final gates.
47.2 done docs/plans/todo-47.2-plugin-token-and-compatibility-boundaries.md Reject remaining negated/non-integer volume tokens, canonicalize integral JSON numbers, preserve fallback state, use standard date-time semantics, and restore positional dataclass compatibility. Named negation/token cases, integral floats, fallback provenance, RFC3339, and positional compatibility pass.
47.3 done docs/plans/todo-47.3-bare-volume-numeric-guard.md Apply malformed numeric rejection to the existing bare volume to, volume at, and volume= syntaxes. Bare decimal/grouped tokens are rejected with no side effect; integer forms remain valid; bounded gate APPROVED.
48 done docs/plans/todo-48-parallel-event-batches.md Execute multiple flat events with bounded parallelism, failure isolation, aggregation, and a terminal barrier. Direct/dual share bounded policy-driven batches with stable order, failure isolation, terminal barrier, and four built-in user flows; full suite passes (372 passed, one existing warning).
48.1 done docs/plans/todo-48.1-batch-canonical-replay-and-terminal-safety.md Canonicalize batch keys, make replay atomic and bounded, guarantee terminal grace after sibling timeout, enforce async timeout around sync handlers, and ensure termination has a farewell. Canonical/in-flight replay, terminal grace, sync timeout responsiveness, farewell, and search two-answer regressions pass.
48.2 done docs/plans/todo-48.2-outcome-cache-and-sync-resource-lifecycle.md Preserve full replay outcomes, generation-scope cleanup, invalid-key isolation, sync worker resource leases, and deduplicated result correlation/policy cardinality. Outcome cache, scope invalidation, deferred sync resource leases, correlation, and single policy messages pass.
48.3 done docs/plans/todo-48.3-strict-canonical-types-and-scope-token-cleanup.md Align canonical-key equality with strict JSON types and replace retained generation tombstones with unique releasable scope tokens. Strict nested JSON types and 1000-cycle scope cleanup pass; final bounded gate APPROVED.
49 done docs/plans/todo-49-tool-mode-ui-session-restore.md Add UI mode selection, workspace/session restoration, and clear EventAgent applicability feedback. Static/WebSocket tests prove requests, snapshots, loaded sessions, and audit context preserve the selected mode.
49.1 done docs/plans/todo-49.1-session-mode-and-atomic-transcript-safety.md Enforce persisted mode, atomically store direct tool rounds, repair incomplete legacy tails, serialize SQLite migration, and hide empty protocol bubbles. Mode mismatches are rejected without config overwrite; disconnect cannot leave unresolved tool calls; concurrent migration succeeds; replay shows only visible assistant content.
49.2 done docs/plans/todo-49.2-session-load-race-and-visible-repair.md Prevent stale session loads from overwriting the selected session and preserve visible text while stripping incomplete tool protocol. Out-of-order loads cannot mutate the active session UI; whitespace-only protocol bubbles stay hidden; incomplete tool rounds retain provider-valid visible content.
49.3 done docs/plans/todo-49.3-stale-replay-token-order.md Prevent a stale replay invocation from invalidating an already-running replay for the current session. A late stale invocation exits before advancing the replay token; the current session still restores normally.
50 done docs/plans/todo-50-comparable-model-tool-metrics.md Record mode-aware model/tool spans and include EventAgent usage so both architectures can be compared fairly. Persistence/API tests report agent, call kind, tokens, TTFT, tool latency, fallback usage, and total turn wall time without double counting.
50.1 done docs/plans/todo-50.1-metric-key-migration-dedup.md Deduplicate legacy metric keys before creating the idempotency index so existing experiment databases remain openable. Migration keeps the earliest row for each duplicate non-null session/metric key, creates the unique index, and remains safe under concurrent initialization.
51 done docs/plans/todo-51-comparison-matrix-closeout.md Add repeatable comparison fixtures, update project documentation, and close the experimental implementation. Both modes pass the same 12-run scenario matrix; focused (231 passed) and full (417 passed) tests, JS syntax, app import, and documentation checks pass.
51.1 done docs/plans/todo-51.1-first-reply-and-protocol-matrix.md Require fast visible first replies for event scenarios and prove each mode uses its intended capability protocol surface. Matrix asserts first reply ordering, no unintended second reply, dual compact event catalog/no provider tools, direct enabled schemas/no generated event catalog, and correct event source.
52 in_progress docs/plans/todo-52-agent-benchmark-runner.md Add a configurable in-process benchmark CLI with deterministic mock cases and live OpenAI-compatible execution. Mock/live runs compare both modes, measure provider/visible TTFT, wall time, tokens and event correctness, write secret-safe JSON/Markdown reports, and use stable exit codes.
52.1 done docs/plans/todo-52.1-benchmark-url-control-characters.md Reject non-printable/control characters in benchmark target URLs before constructing the HTTP client. Strict config tests cover NUL, CR/LF, tabs, and other Unicode/control characters while valid HTTP(S) URLs remain accepted.
52.2 done docs/plans/todo-52.2-benchmark-url-transport-compatibility.md Validate benchmark target hostname/URL compatibility with the production HTTP transport at the config boundary. Invalid IDNA/port/transport URLs are rejected as config errors while valid Unicode hostnames accepted by the transport remain valid.
52.3 done docs/plans/todo-52.3-benchmark-case-mode-single-source.md Make benchmark case IDs and invocation modes single-source enums shared by config, catalog, runner, and tests. Adding or removing an enum member cannot leave config literals, exported choices, and catalog keys silently inconsistent.
52.4 done docs/plans/todo-52.4-benchmark-visible-ttft-and-call-counts.md Measure the first non-whitespace visible content and enforce each case's expected ChatAgent/fallback call counts. Blank deltas cannot reduce visible TTFT; extra or missing model/fallback calls produce semantic failures even when timing and ledger agree.
52.5 done docs/plans/todo-52.5-benchmark-ledger-event-identity.md Cross-check detected, batch-result, and tool-metric event identities and preserve attempted model counts on failed runs. A successful result from the wrong event cannot pass; failed streams still report started ChatAgent/fallback attempts from timing records.
52.6 done docs/plans/todo-52.6-benchmark-report-auditability.md Complete report latency coverage, concurrent pair reservation, runner-level error auditability, and byte-bounded errors. Reports aggregate model/tool latency, concurrent writers cannot mix pairs, catastrophic runner errors appear in report totals/details, zero samples warn, and errors stay within 2 KiB UTF-8.
52.7 done docs/plans/todo-52.7-benchmark-output-reservation-and-cardinality.md Prevent report overwrite/stale reservations, suppress argparse value echoes, and reject missing or extra benchmark results. Final paths are atomically reserved without lock files, malformed CLI values are not echoed, and actual run cardinality must equal cases × modes × iterations.
52.8 done docs/plans/todo-52.8-benchmark-no-overwrite-publication.md Publish completed report files with atomic no-overwrite hard links instead of check-then-replace ownership logic. A destination created at any point wins without being overwritten; the writer retries another suffix or fails cleanly with no mixed pair.
52.9 done docs/plans/todo-52.9-atomic-report-bundles-and-audit-parity.md Replace two-path report publication with one atomic bundle boundary and close report/CLI audit parity gaps. Concurrent writers publish complete bundles without rollback deletion races; Markdown preserves all failure facts and output errors map to exit code 2.
52.10 done docs/plans/todo-52.10-benchmark-bundle-cli-contract.md Align CLI report discovery tests with the atomically published bundle layout. CLI tests consume the reported bundle paths without restoring root-level compatibility copies or weakening redaction, exit-code, and cardinality checks.
52.11 done docs/plans/todo-52.11-bundle-interrupt-and-path-safety.md Preserve complete bundles across post-symlink interrupts and reject unsafe report timestamp path components. Interruptions cannot leave dangling public bundles; explicit timestamps stay inside output_dir; fsync tests verify both files and both directories.