agent-lab

Agent Lab

Agent Lab is a FastAPI debug console for comparing ChatAgent capability-invocation protocols on one shared event runtime. The implementation uses a generic event kernel plus a flat registry of built-in plugins; invocation mode changes how the model selects capabilities, not how accepted events are normalized, validated, scheduled, executed, persisted, or reported.

The architecture source of truth is the production events design. The repeatable experiment and closeout scope is tracked by Todo 51.

Architecture and Built-ins

The generic kernel owns event lookup, deterministic resolution, validation, bounded fallback, deduplication, conflict-aware concurrency, timeouts, terminal barriers, normalized results, result-policy aggregation, audit events, and metrics. Runtime and the kernel do not branch on a built-in event name.

The four production-shaped plugins are flat, namespaced definitions:

Event	Result policy	Current default behavior
session.terminate	terminate	Records an in-memory terminated result and ends the debug session on success.
device.volume.adjust	silent_success	Validates and records an in-memory volume action; success adds no second answer.
calendar.schedule.create	template_follow_up	Records an in-memory schedule and emits a deterministic confirmation.
knowledge.web.search	llm_follow_up	Returns deterministic mock sources, then asks ChatAgent for one grounded follow-up.

The default registry also retains the lab-only handoff_note, mock_search, and mock_ticket tools. GET /api/tools is the authoritative runtime catalog.

Invocation Modes

Set tool_invocation_mode per request:

Mode	Selection protocol	Argument source	Mode-specific cost or dependency
dual_agent	ChatAgent receives a compact catalog and emits hidden text events after visible content.	Shared deterministic resolvers run first; EventAgent may make a bounded fallback model call for incomplete supported events.	Depends on marker adherence and adds fallback calls when deterministic resolution is incomplete.
chat_agent_tools	ChatAgent receives enabled provider tool schemas and emits native tool calls.	ChatAgent proposes arguments; the shared kernel still normalizes, validates, authorizes, deduplicates, schedules, and executes them.	Full schemas and provider tool transcript remain in ChatAgent context and usage; EventAgent model/prompt settings are inactive for provider-resolved calls.

Both modes share the same enabled definitions, JSON schemas, handlers/adapters, event budget, batch concurrency, timeout, conflict and terminal policies, normalized results, and user-visible success/failure contract. Provider-native assistant/tool transcript details are intentionally allowed to differ.

Web search always has two answers

For knowledge.web.search, both modes implement the same visible sequence:

1. ChatAgent emits a useful or qualified first answer.
2. The shared pipeline executes the search and emits its tool result.
3. ChatAgent makes exactly one follow-up model call and emits a distinct second answer.

The first answer must not wait for the search adapter, and the second answer must use the compact returned evidence. This is a shared behavior contract, not a provider transcript normalization trick.

Current Adapter Limits

The default application wiring is safe for local experiments, but it is not production integration:

• session termination is an in-memory result, not a real dialogue-service command;
• volume adjustment does not control a device or publish MQTT/device actions;
• schedule creation does not write to a real calendar or reminder service;
• web search returns deterministic example.invalid mock sources and does not access the internet.

Inject implementations of SessionTerminationPort, DeviceVolumePort, CalendarSchedulePort, and WebSearchPort when testing service-owned adapters. Real device, calendar, session, and internet integration belongs outside the default Agent Lab wiring.

Configure and Run

Install dependencies and create a local environment file:

uv sync
cp .env.example .env

Supported settings use the AGENT_LAB_ prefix:

• AGENT_LAB_OPENAI_API_KEY: API key for the compatible provider;
• AGENT_LAB_OPENAI_BASE_URL: provider /v1 root;
• AGENT_LAB_OPENAI_DEFAULT_MODEL: model used when a request leaves model blank;
• AGENT_LAB_OPENAI_INCLUDE_USAGE: disable if the provider rejects streamed usage options;
• AGENT_LAB_REQUEST_TIMEOUT_SECONDS: provider HTTP timeout;
• AGENT_LAB_DATABASE_PATH: SQLite session/audit/usage database path (defaults to .agent_lab.sqlite3).

Do not commit secrets. Start the app with:

uv run uvicorn agent_lab.main:app --reload

Open http://127.0.0.1:8000. The UI can select either invocation mode, enable tools/events, configure ChatAgent and applicable EventAgent settings, set event loop/concurrency/batch limits, create or restore sessions, replay messages and audit data, and inspect per-call, per-turn, and per-session usage.

Benchmark CLI

agent-lab-benchmark runs the same built-in scenarios through dual_agent and chat_agent_tools, then writes matching machine-readable and review-friendly reports. It runs in process against the production runtime and event kernel; it does not start FastAPI or add WebSocket/network timing to the measured turn.

The API key is read only from .env or the environment through AGENT_LAB_OPENAI_API_KEY. Do not put a key in the benchmark JSON or commit it. The JSON config supplies the OpenAI-compatible base_url, model, run count, modes, and cases:

cp benchmarks/single-target.example.json benchmarks/single-target.local.json

Edit the copied file before a live run. The checked-in example deliberately uses https://provider.example/v1 and replace-with-model-id placeholders and contains no secret.

Start with the deterministic mock benchmark. This command does not require an API key and makes no network request:

uv run agent-lab-benchmark \
  --config benchmarks/single-target.example.json \
  --mock

For a live OpenAI-compatible provider, set the key locally, configure a real base_url and model in the copied JSON, and omit --mock:

export AGENT_LAB_OPENAI_API_KEY='replace-locally'
uv run agent-lab-benchmark \
  --config benchmarks/single-target.local.json

CLI values override JSON values. --base-url, --model, and --runs replace their scalar settings; repeat --case or --mode to replace the corresponding JSON list. For example, this still performs a zero-network mock run:

uv run agent-lab-benchmark \
  --config benchmarks/single-target.example.json \
  --mock \
  --runs 3 \
  --case web_search_two_answers \
  --mode dual_agent \
  --mode chat_agent_tools

The six built-in cases are:

Case	Expected behavior
ordinary_chat	One visible answer and no event.
device_volume_silent	One volume event with silent success and no model follow-up.
calendar_schedule_template	One schedule event followed by the deterministic confirmation template.
web_search_two_answers	A fast first answer, one search event, and exactly one grounded second answer.
session_terminate	One farewell, one terminate event, and a terminal session.
parallel_volume_schedule	Volume and schedule events both succeed and their async handlers are proven to overlap.

Each run records provider TTFT (first provider stream item), visible TTFT (first non-whitespace user-visible delta), turn wall time, per-call model latency, tool handler latency, prompt/completion/total/cached tokens, model/fallback/tool call counts, and event names, sources, statuses, and semantic correctness. Provider TTFT and visible TTFT are intentionally separate; model and tool spans may overlap and are not added to manufacture a total latency.

Per-call model elapsed time accumulates only time awaiting the provider stream iterator. It excludes downstream runtime work and consumer backpressure after a yield, while provider TTFT and visible TTFT remain wall-clock arrival metrics. Token totals are reported only when every model and argument-fallback call returns usage; otherwise all token metrics are null and the run fails with an explicit missing-usage semantic error. The parallel volume/schedule case uses a benchmark-owned barrier probe and passes only when both event handlers are active at the same time, not merely when two successful event results exist.

By default, reports are published as one complete bundle:

outputs/benchmarks/<timestamp>/report.json
outputs/benchmarks/<timestamp>/report.md

The timestamp path is a single relative symlink published only after both files are fully written and synced; collisions receive a numeric suffix. JSON is the machine-readable source of truth and Markdown is rendered from the same report model.

Exit codes are stable:

• 0: all runs passed and reports were written;
• 1: at least one request or semantic check failed, but reports were written;
• 2: CLI, config, environment, runner-cardinality, or report/output failure;
• 130: interruption.

For a fair mode comparison, keep the provider/model, generation parameters, input cases, run count, enabled events, limits, timeouts, adapters, and run conditions unchanged. Use fresh paired runs and inspect failed samples instead of comparing only averages. p95 is reported with a low-confidence warning when fewer than 20 successful samples exist in a case/mode group.

Interpret results within these limits:

• mock mode and the repository's scripted tests validate deterministic flow, event calls, metrics, and report generation, not live provider behavior;
• the runner is in process, so it does not measure FastAPI, WebSocket, service transport, reconnect, or deployment overhead;
• default event adapters are in-memory implementations and do not control a device, create a real calendar entry, terminate a service conversation, or search the public web;
• the CLI does not calculate pricing or automatically declare/rank a winner; evaluate latency, tokens, correctness, and operational tradeoffs together;
• live runs send prompts to the configured provider and may incur API charges.

WebSocket

Connect to ws://127.0.0.1:8000/ws/debug. The smallest valid initial payload is:

{
  "user_message": "Hello",
  "chat_agent": {}
}

This uses the default dual_agent mode and default EventAgent configuration. A minimal built-in event request can make the comparison controls explicit:

{
  "user_message": "Set the device volume to 30",
  "chat_agent": {},
  "event_agent": {
    "enabled_tools": ["device.volume.adjust"]
  },
  "tool_invocation_mode": "dual_agent"
}

Change only tool_invocation_mode to chat_agent_tools for the other protocol while keeping model parameters, enabled events, event limits, input, and adapter wiring equal. After session_started, the same socket accepts later turns as {"type":"user_message","content":"..."} until a terminal done or error.

ws_smoke.py is not a dual-mode benchmark

With the server running, scripts/ws_smoke.py sends one convenience request and prints streamed JSON:

uv run python scripts/ws_smoke.py --message "Debug this agent handoff path."

It defaults to dual_agent, exercises the handoff_note path, has no paired mode runner or fairness controls, and does not collect the Todo 51 scenario matrix. Use it only for WebSocket connectivity and payload smoke testing.

HTTP API

Method and path	Purpose
GET /health	Process health.
GET /	Static debug UI.
GET /api/tools	Current flat registry catalog and schemas.
POST /api/sessions	Create a persisted session with optional title and config.
GET /api/sessions	List persisted sessions.
GET /api/sessions/{session_id}	Read session metadata and immutable comparison config.
GET /api/sessions/{session_id}/messages	Replay persisted visible/provider messages.
GET /api/sessions/{session_id}/audit	Read ordered runtime audit events.
GET /api/sessions/{session_id}/usage	Read comparable call, turn, and session metrics.

/usage Fair-Comparison Contract

GET /api/sessions/{session_id}/usage returns calls, turns, and session. Interpret it with these controls:

• compare runs with the same model parameters, enabled event set, user input, output budget, registry/adapters, event budget, concurrency, and timeouts;
• token totals include real ChatAgent and EventAgent model calls only; tool execution rows have zero tokens;
• direct-mode provider tool schema and assistant/tool protocol tokens remain counted rather than being normalized away;
• EventAgent argument-fallback calls are separate argument_fallback rows and contribute their model tokens;
• turn_wall_time_ms is the turn/session wall-clock measure. Do not add model elapsed_ms or overlapping tool_latency_ms spans to estimate total latency;
• only logical tool executions count: deduplicated/replayed clones do not inflate tool_count or fallback count;
• call-level ttft_ms measures the first provider stream item. Visible-text TTFT is a separate runtime/audit observation and must not be conflated with it.

Verification

Run the closeout matrix and focused runtime checks:

uv run pytest tests/test_tool_invocation_comparison.py -q
uv run pytest tests/test_debug_runtime.py tests/test_builtin_event_plugins.py tests/test_sqlite_store.py tests/test_websocket_api.py -q

Run the repository gates:

uv run pytest
node --check src/agent_lab/presentation/static/app.js
uv run python -c "from agent_lab.main import app; print(app.title)"
git diff --check

These tests establish repeatable application semantics with scripted model clients and injected/default mock ports. They do not prove live-provider content plus-tool streaming behavior, real network search quality, real device/calendar side effects, production credentials, or production latency. Run provider and service integration smokes separately before drawing deployment conclusions.