# Agent Lab

Agent Lab is a FastAPI debug console for comparing ChatAgent capability-invocation
protocols on one shared event runtime. The implementation uses a generic event
kernel plus a flat registry of built-in plugins; invocation mode changes how the
model selects capabilities, not how accepted events are normalized, validated,
scheduled, executed, persisted, or reported.

The architecture source of truth is the
[production events design](docs/plans/2026-07-12-chat-event-agent-production-events-design.md).
The repeatable experiment and closeout scope is tracked by
[Todo 51](docs/plans/todo-51-comparison-matrix-closeout.md).

## Architecture and Built-ins

The generic kernel owns event lookup, deterministic resolution, validation,
bounded fallback, deduplication, conflict-aware concurrency, timeouts, terminal
barriers, normalized results, result-policy aggregation, audit events, and
metrics. Runtime and the kernel do not branch on a built-in event name.

The four production-shaped plugins are flat, namespaced definitions:

| Event | Result policy | Current default behavior |
| --- | --- | --- |
| `session.terminate` | `terminate` | Records an in-memory terminated result and ends the debug session on success. |
| `device.volume.adjust` | `silent_success` | Validates and records an in-memory volume action; success adds no second answer. |
| `calendar.schedule.create` | `template_follow_up` | Records an in-memory schedule and emits a deterministic confirmation. |
| `knowledge.web.search` | `llm_follow_up` | Returns deterministic mock sources, then asks ChatAgent for one grounded follow-up. |

The default registry also retains the lab-only `handoff_note`, `mock_search`, and
`mock_ticket` tools. `GET /api/tools` is the authoritative runtime catalog.

## Invocation Modes

Set `tool_invocation_mode` per request:

| Mode | Selection protocol | Argument source | Mode-specific cost or dependency |
| --- | --- | --- | --- |
| `dual_agent` | ChatAgent receives a compact catalog and emits hidden text events after visible content. | Shared deterministic resolvers run first; EventAgent may make a bounded fallback model call for incomplete supported events. | Depends on marker adherence and adds fallback calls when deterministic resolution is incomplete. |
| `chat_agent_tools` | ChatAgent receives enabled provider tool schemas and emits native tool calls. | ChatAgent proposes arguments; the shared kernel still normalizes, validates, authorizes, deduplicates, schedules, and executes them. | Full schemas and provider tool transcript remain in ChatAgent context and usage; EventAgent model/prompt settings are inactive for provider-resolved calls. |

Both modes share the same enabled definitions, JSON schemas, handlers/adapters,
event budget, batch concurrency, timeout, conflict and terminal policies,
normalized results, and user-visible success/failure contract. Provider-native
assistant/tool transcript details are intentionally allowed to differ.

### Web search always has two answers

For `knowledge.web.search`, both modes implement the same visible sequence:

1. ChatAgent emits a useful or qualified first answer.
2. The shared pipeline executes the search and emits its tool result.
3. ChatAgent makes exactly one follow-up model call and emits a distinct second answer.

The first answer must not wait for the search adapter, and the second answer must
use the compact returned evidence. This is a shared behavior contract, not a
provider transcript normalization trick.

## Current Adapter Limits

The default application wiring is safe for local experiments, but it is not
production integration:

- session termination is an in-memory result, not a real dialogue-service command;
- volume adjustment does not control a device or publish MQTT/device actions;
- schedule creation does not write to a real calendar or reminder service;
- web search returns deterministic `example.invalid` mock sources and does not access the internet.

Inject implementations of `SessionTerminationPort`, `DeviceVolumePort`,
`CalendarSchedulePort`, and `WebSearchPort` when testing service-owned adapters.
Real device, calendar, session, and internet integration belongs outside the
default Agent Lab wiring.

## Configure and Run

Install dependencies and create a local environment file:

```bash
uv sync
cp .env.example .env
```

Supported settings use the `AGENT_LAB_` prefix:

- `AGENT_LAB_OPENAI_API_KEY`: API key for the compatible provider;
- `AGENT_LAB_OPENAI_BASE_URL`: provider `/v1` root;
- `AGENT_LAB_OPENAI_DEFAULT_MODEL`: model used when a request leaves `model` blank;
- `AGENT_LAB_OPENAI_INCLUDE_USAGE`: disable if the provider rejects streamed usage options;
- `AGENT_LAB_REQUEST_TIMEOUT_SECONDS`: provider HTTP timeout;
- `AGENT_LAB_DATABASE_PATH`: SQLite session/audit/usage database path (defaults to `.agent_lab.sqlite3`).

Do not commit secrets. Start the app with:

```bash
uv run uvicorn agent_lab.main:app --reload
```

Open `http://127.0.0.1:8000`. The UI can select either invocation mode, enable
tools/events, configure ChatAgent and applicable EventAgent settings, set event
loop/concurrency/batch limits, create or restore sessions, replay messages and
audit data, and inspect per-call, per-turn, and per-session usage.

## WebSocket

Connect to `ws://127.0.0.1:8000/ws/debug`. The smallest valid initial payload is:

```json
{
  "user_message": "Hello",
  "chat_agent": {}
}
```

This uses the default `dual_agent` mode and default EventAgent configuration. A
minimal built-in event request can make the comparison controls explicit:

```json
{
  "user_message": "Set the device volume to 30",
  "chat_agent": {},
  "event_agent": {
    "enabled_tools": ["device.volume.adjust"]
  },
  "tool_invocation_mode": "dual_agent"
}
```

Change only `tool_invocation_mode` to `chat_agent_tools` for the other protocol
while keeping model parameters, enabled events, event limits, input, and adapter
wiring equal. After `session_started`, the same socket accepts later turns as
`{"type":"user_message","content":"..."}` until a terminal `done` or `error`.

### `ws_smoke.py` is not a dual-mode benchmark

With the server running, `scripts/ws_smoke.py` sends one convenience request and
prints streamed JSON:

```bash
uv run python scripts/ws_smoke.py --message "Debug this agent handoff path."
```

It defaults to `dual_agent`, exercises the `handoff_note` path, has no paired
mode runner or fairness controls, and does not collect the Todo 51 scenario
matrix. Use it only for WebSocket connectivity and payload smoke testing.

## HTTP API

| Method and path | Purpose |
| --- | --- |
| `GET /health` | Process health. |
| `GET /` | Static debug UI. |
| `GET /api/tools` | Current flat registry catalog and schemas. |
| `POST /api/sessions` | Create a persisted session with optional `title` and `config`. |
| `GET /api/sessions` | List persisted sessions. |
| `GET /api/sessions/{session_id}` | Read session metadata and immutable comparison config. |
| `GET /api/sessions/{session_id}/messages` | Replay persisted visible/provider messages. |
| `GET /api/sessions/{session_id}/audit` | Read ordered runtime audit events. |
| `GET /api/sessions/{session_id}/usage` | Read comparable call, turn, and session metrics. |

## `/usage` Fair-Comparison Contract

`GET /api/sessions/{session_id}/usage` returns `calls`, `turns`, and `session`.
Interpret it with these controls:

- compare runs with the same model parameters, enabled event set, user input,
  output budget, registry/adapters, event budget, concurrency, and timeouts;
- token totals include real ChatAgent and EventAgent model calls only; tool
  execution rows have zero tokens;
- direct-mode provider tool schema and assistant/tool protocol tokens remain
  counted rather than being normalized away;
- EventAgent argument-fallback calls are separate `argument_fallback` rows and
  contribute their model tokens;
- `turn_wall_time_ms` is the turn/session wall-clock measure. Do not add model
  `elapsed_ms` or overlapping `tool_latency_ms` spans to estimate total latency;
- only logical tool executions count: deduplicated/replayed clones do not inflate
  `tool_count` or fallback count;
- call-level `ttft_ms` measures the first provider stream item. Visible-text TTFT
  is a separate runtime/audit observation and must not be conflated with it.

## Verification

Run the closeout matrix and focused runtime checks:

```bash
uv run pytest tests/test_tool_invocation_comparison.py -q
uv run pytest tests/test_debug_runtime.py tests/test_builtin_event_plugins.py tests/test_sqlite_store.py tests/test_websocket_api.py -q
```

Run the repository gates:

```bash
uv run pytest
node --check src/agent_lab/presentation/static/app.js
uv run python -c "from agent_lab.main import app; print(app.title)"
git diff --check
```

These tests establish repeatable application semantics with scripted model
clients and injected/default mock ports. They do not prove live-provider content
plus-tool streaming behavior, real network search quality, real device/calendar
side effects, production credentials, or production latency. Run provider and
service integration smokes separately before drawing deployment conclusions.