agent-lab

Agent Lab

Agent Lab is a FastAPI debug console for comparing ChatAgent capability-invocation protocols on one shared event runtime. The implementation uses a generic event kernel plus a flat registry of built-in plugins; invocation mode changes how the model selects capabilities, not how accepted events are normalized, validated, scheduled, executed, persisted, or reported.

The architecture source of truth is the production events design. The repeatable experiment and closeout scope is tracked by Todo 51.

Architecture and Built-ins

The generic kernel owns event lookup, deterministic resolution, validation, bounded fallback, deduplication, conflict-aware concurrency, timeouts, terminal barriers, normalized results, result-policy aggregation, audit events, and metrics. Runtime and the kernel do not branch on a built-in event name.

The four production-shaped plugins are flat, namespaced definitions:

Event	Result policy	Current default behavior
session.terminate	terminate	Records an in-memory terminated result and ends the debug session on success.
device.volume.adjust	silent_success	Validates and records an in-memory volume action; success adds no second answer.
calendar.schedule.create	template_follow_up	Records an in-memory schedule and emits a deterministic confirmation.
knowledge.web.search	llm_follow_up	Returns deterministic mock sources, then asks ChatAgent for one grounded follow-up.

The default registry also retains the lab-only handoff_note, mock_search, and mock_ticket tools. GET /api/tools is the authoritative runtime catalog.

Invocation Modes

Set tool_invocation_mode per request:

Mode	Selection protocol	Argument source	Mode-specific cost or dependency
dual_agent	ChatAgent receives a compact catalog and emits hidden text events after visible content.	Shared deterministic resolvers run first; EventAgent may make a bounded fallback model call for incomplete supported events.	Depends on marker adherence and adds fallback calls when deterministic resolution is incomplete.
chat_agent_tools	ChatAgent receives enabled provider tool schemas and emits native tool calls.	ChatAgent proposes arguments; the shared kernel still normalizes, validates, authorizes, deduplicates, schedules, and executes them.	Full schemas and provider tool transcript remain in ChatAgent context and usage; EventAgent model/prompt settings are inactive for provider-resolved calls.

Both modes share the same enabled definitions, JSON schemas, handlers/adapters, event budget, batch concurrency, timeout, conflict and terminal policies, normalized results, and user-visible success/failure contract. Provider-native assistant/tool transcript details are intentionally allowed to differ.

Web search always has two answers

For knowledge.web.search, both modes implement the same visible sequence:

1. ChatAgent emits a useful or qualified first answer.
2. The shared pipeline executes the search and emits its tool result.
3. ChatAgent makes exactly one follow-up model call and emits a distinct second answer.

The first answer must not wait for the search adapter, and the second answer must use the compact returned evidence. This is a shared behavior contract, not a provider transcript normalization trick.

Current Adapter Limits

The default application wiring is safe for local experiments, but it is not production integration:

• session termination is an in-memory result, not a real dialogue-service command;
• volume adjustment does not control a device or publish MQTT/device actions;
• schedule creation does not write to a real calendar or reminder service;
• web search returns deterministic example.invalid mock sources and does not access the internet.

Inject implementations of SessionTerminationPort, DeviceVolumePort, CalendarSchedulePort, and WebSearchPort when testing service-owned adapters. Real device, calendar, session, and internet integration belongs outside the default Agent Lab wiring.

Configure and Run

Install dependencies and create a local environment file:

uv sync
cp .env.example .env

Supported settings use the AGENT_LAB_ prefix:

• AGENT_LAB_OPENAI_API_KEY: API key for the compatible provider;
• AGENT_LAB_OPENAI_BASE_URL: provider /v1 root;
• AGENT_LAB_OPENAI_DEFAULT_MODEL: model used when a request leaves model blank;
• AGENT_LAB_OPENAI_INCLUDE_USAGE: disable if the provider rejects streamed usage options;
• AGENT_LAB_REQUEST_TIMEOUT_SECONDS: provider HTTP timeout;
• AGENT_LAB_DATABASE_PATH: SQLite session/audit/usage database path (defaults to .agent_lab.sqlite3).

Do not commit secrets. Start the app with:

uv run uvicorn agent_lab.main:app --reload

Open http://127.0.0.1:8000. The UI can select either invocation mode, enable tools/events, configure ChatAgent and applicable EventAgent settings, set event loop/concurrency/batch limits, create or restore sessions, replay messages and audit data, and inspect per-call, per-turn, and per-session usage.

WebSocket

Connect to ws://127.0.0.1:8000/ws/debug. The smallest valid initial payload is:

{
  "user_message": "Hello",
  "chat_agent": {}
}

This uses the default dual_agent mode and default EventAgent configuration. A minimal built-in event request can make the comparison controls explicit:

{
  "user_message": "Set the device volume to 30",
  "chat_agent": {},
  "event_agent": {
    "enabled_tools": ["device.volume.adjust"]
  },
  "tool_invocation_mode": "dual_agent"
}

Change only tool_invocation_mode to chat_agent_tools for the other protocol while keeping model parameters, enabled events, event limits, input, and adapter wiring equal. After session_started, the same socket accepts later turns as {"type":"user_message","content":"..."} until a terminal done or error.

ws_smoke.py is not a dual-mode benchmark

With the server running, scripts/ws_smoke.py sends one convenience request and prints streamed JSON:

uv run python scripts/ws_smoke.py --message "Debug this agent handoff path."

It defaults to dual_agent, exercises the handoff_note path, has no paired mode runner or fairness controls, and does not collect the Todo 51 scenario matrix. Use it only for WebSocket connectivity and payload smoke testing.

HTTP API

Method and path	Purpose
GET /health	Process health.
GET /	Static debug UI.
GET /api/tools	Current flat registry catalog and schemas.
POST /api/sessions	Create a persisted session with optional title and config.
GET /api/sessions	List persisted sessions.
GET /api/sessions/{session_id}	Read session metadata and immutable comparison config.
GET /api/sessions/{session_id}/messages	Replay persisted visible/provider messages.
GET /api/sessions/{session_id}/audit	Read ordered runtime audit events.
GET /api/sessions/{session_id}/usage	Read comparable call, turn, and session metrics.

/usage Fair-Comparison Contract

GET /api/sessions/{session_id}/usage returns calls, turns, and session. Interpret it with these controls:

• compare runs with the same model parameters, enabled event set, user input, output budget, registry/adapters, event budget, concurrency, and timeouts;
• token totals include real ChatAgent and EventAgent model calls only; tool execution rows have zero tokens;
• direct-mode provider tool schema and assistant/tool protocol tokens remain counted rather than being normalized away;
• EventAgent argument-fallback calls are separate argument_fallback rows and contribute their model tokens;
• turn_wall_time_ms is the turn/session wall-clock measure. Do not add model elapsed_ms or overlapping tool_latency_ms spans to estimate total latency;
• only logical tool executions count: deduplicated/replayed clones do not inflate tool_count or fallback count;
• call-level ttft_ms measures the first provider stream item. Visible-text TTFT is a separate runtime/audit observation and must not be conflated with it.

Verification

Run the closeout matrix and focused runtime checks:

uv run pytest tests/test_tool_invocation_comparison.py -q
uv run pytest tests/test_debug_runtime.py tests/test_builtin_event_plugins.py tests/test_sqlite_store.py tests/test_websocket_api.py -q

Run the repository gates:

uv run pytest
node --check src/agent_lab/presentation/static/app.js
uv run python -c "from agent_lab.main import app; print(app.title)"
git diff --check

These tests establish repeatable application semantics with scripted model clients and injected/default mock ports. They do not prove live-provider content plus-tool streaming behavior, real network search quality, real device/calendar side effects, production credentials, or production latency. Run provider and service integration smokes separately before drawing deployment conclusions.