agent-lab/docs/plans/2026-07-13-agent-benchmark-design.md

Agent Invocation Benchmark Design

Status: approved for implementation by the user's "feasible means proceed" instruction

Date: 2026-07-13

Goal

Provide a repeatable benchmark command where the operator configures an OpenAI-compatible base_url, model, and API key, then automatically runs the same business scenarios in dual_agent and chat_agent_tools modes and produces reviewable evaluation reports.

Chosen Architecture

The first version is an in-process packaged CLI. It constructs the existing OpenAICompatibleChatClient, DebugRuntime, generic event kernel, built-in mock adapters, and SQLite session store directly. It does not start FastAPI or connect through WebSocket.

This preserves the complete ChatAgent/EventAgent/event-kernel behavior while reducing benchmark orchestration noise. A later black-box benchmark may exercise WebSocket/network overhead after the in-process baseline is stable.

Alternatives considered:

• WebSocket black-box benchmark: validates the complete service boundary but adds server lifecycle, port, reconnect, and transport timing noise.
• Provider-only benchmark: easiest latency measurement but cannot validate event detection, fallback, execution, result policies, or two-answer behavior.

Configuration

The API key is read only from .env/environment through AGENT_LAB_OPENAI_API_KEY. Benchmark JSON must not contain secrets.

{
  "schema_version": 1,
  "base_url": "https://provider.example/v1",
  "model": "model-name",
  "runs_per_case": 1,
  "modes": ["dual_agent", "chat_agent_tools"],
  "cases": [
    "ordinary_chat",
    "device_volume_silent",
    "calendar_schedule_template",
    "web_search_two_answers",
    "session_terminate",
    "parallel_volume_schedule"
  ]
}

CLI overrides JSON using CLI > JSON. Repeatable --case and --mode options replace the corresponding JSON lists. Live mode requires explicit base URL, model, and API key. --mock requires no API key and makes no network request.

The first version intentionally supports one target per command. Multi-provider orchestration, parallel load, custom case DSL, pricing, CSV, and automatic winner ranking are out of scope.

Execution and Isolation

Each case/mode/iteration run receives:

• a unique session ID;
• a dedicated temporary SQLite database;
• the production DebugRuntime and default built-in in-memory adapters;
• a timing wrapper around the configured chat client;
• identical model parameters, enabled event definitions, event limits, and inputs across modes.

The runner uses start_session() and consumes output through the first turn_completed or error, then reads persisted audit and usage data. It always closes the runtime/client and removes the temporary database.

One failed run does not stop later combinations. Results preserve deterministic case → mode → iteration ordering.

Built-in Cases and Semantic Checks

The benchmark catalog owns the same six production-shaped cases as the closeout matrix:

• ordinary chat: one visible answer, no event;
• device volume: fast first acknowledgement, one successful device.volume.adjust, no model follow-up;
• schedule: fast acknowledgement, one successful calendar.schedule.create, one deterministic template;
• web search: first answer, successful knowledge.web.search, exactly one second answer;
• terminate: one farewell, successful session.terminate, terminal session;
• parallel volume plus schedule: both events accepted, stable result order, one schedule template.

Mock mode emits deterministic streams for these cases. Live mode relies on the configured model and marks semantic mismatches as failed runs rather than silently accepting them.

Checks include answer count, first-answer ordering before tool completion, exact expected event names/source, normalized event status, model/tool/fallback counts, and terminal behavior.

Metrics

Each run records:

• initial provider TTFT from the first yielded provider stream item;
• first visible delta TTFT measured from benchmark turn start;
• pure model call elapsed values from the timing wrapper;
• turn_wall_time_ms from the persisted turn ledger;
• prompt, completion, total, and cached tokens;
• ChatAgent model call count, EventAgent fallback count, logical tool count;
• detected event names/sources, tool statuses, and handler latency;
• semantic failures and a redacted/truncated error.

Provider TTFT, visible TTFT, pure model elapsed, tool latency, and turn wall time remain separate. Overlapping spans are never summed to manufacture a total.

For each case/mode group, successful non-null samples produce p50 and p95 using linear interpolation with index = (n - 1) * q. Markdown flags p95 as low confidence when fewer than 20 successful samples exist.

Reports and Exit Codes

One command writes matching timestamped files:

• outputs/benchmarks/<timestamp>.json as the machine-readable source of truth;
• outputs/benchmarks/<timestamp>.md rendered from the same report model.

Exit codes:

• 0: every run passed and both reports were written;
• 1: at least one request or semantic check failed, but reports were written;
• 2: CLI/config/environment/report-write failure;
• 130: interruption.

Security

• API keys cannot appear in JSON configuration or report models.
• Base URLs containing userinfo, query, or fragment are rejected.
• Errors are redacted and truncated before stdout, JSON, or Markdown rendering.
• Redaction covers configured key values, Bearer/API-Key forms, and common key/token parameters.
• Reports do not include headers, raw SSE chunks, full provider bodies, or secrets.
• The default benchmark adapters do not reach real device, calendar, session, or web-search services.

Testing

• Config/CLI override and validation tests.
• Deterministic mock end-to-end run with zero network construction.
• Metric source separation and semantic validation tests.
• Partial-failure continuation and exit-code tests.
• Percentile, aggregation, JSON/Markdown consistency, atomic write, and secret-redaction tests.
• Full repository regression and one generated mock report smoke.