# ChatAgent / EventAgent Production Events Design

**Status:** approved design

**Date:** 2026-07-12

## 1. Goal

Evaluate two configurable capability-invocation architectures for the service voice-reply path, primarily targeting a low-latency conversational toy:

- `ChatAgent` owns low-latency, natural, user-facing speech.
- the shared event pipeline owns extensible argument resolution, validation, concurrency, execution, and result policy;
- in dual-agent mode ChatAgent selects event names and EventAgent supplies missing-argument fallback, while direct mode lets ChatAgent select events and propose arguments through native tool calls.

The first production events are:

- terminate the current voice session;
- adjust device volume;
- add a calendar/reminder schedule;
- search the web and provide a second, search-grounded answer.

The primary success criterion is to measure whether keeping provider tool schemas off ChatAgent's first-token critical path actually improves reply latency and cost without reducing event accuracy. The lab must support both direct ChatAgent tool calling and the dual-agent text-event design under one consistent capability contract.

## 2. Architecture Decision

Keep one shared flat event registry and make the event-selection path configurable. The two experimental modes differ only in how events are selected and how initial arguments are proposed; they share argument normalization, execution, concurrency, result policies, and user-visible behavior.

Configuration uses one explicit `tool_invocation_mode` value:

- `dual_agent`: ChatAgent receives a compact event catalog but no provider `tools`. It streams its visible reply and appends internal event names in the text-event block. EventAgent resolves missing arguments.
- `chat_agent_tools`: ChatAgent receives the enabled provider tool schemas and emits native tool calls, including initial arguments. The shared event pipeline validates and normalizes those arguments before execution.

In `dual_agent` mode, the parser hides the event block from the user and dispatches the named events to EventAgent. In `chat_agent_tools` mode, the streaming client collects native tool-call deltas while continuing to expose any visible assistant content allowed by the provider.

Both modes convert model output into the same internal event batch. The shared pipeline performs deterministic parsing and normalization first, then a small LLM fallback only when required fields remain ambiguous or missing. It validates the resolved arguments and invokes the same handlers through the registry.

ChatAgent may emit multiple event names in one response. Independent events are resolved and executed concurrently. The runtime applies an explicit conflict policy before execution so terminal or mutually exclusive side effects do not race unpredictably.

Each event definition declares a result policy so the runtime does not hardcode behavior by event name:

| Result policy | Runtime behavior |
| --- | --- |
| `terminate` | Execute after ChatAgent's farewell, end the session on success, and do not run ChatAgent again. |
| `silent_success` | Keep ChatAgent's first acknowledgement as the normal response; run a corrective follow-up only if execution fails. |
| `template_follow_up` | Produce a deterministic confirmation from the normalized execution result without another LLM call. |
| `llm_follow_up` | Feed a compact structured result back to ChatAgent and produce a second user-facing answer. |

Initial mapping:

| Event | Result policy |
| --- | --- |
| `terminate_session` | `terminate` |
| `adjust_device_volume` | `silent_success` |
| `add_schedule` | `template_follow_up` |
| `web_search` | `llm_follow_up` |

### 2.1 Experimental consistency contract

The comparison must not become two separate implementations. Both modes use:

- the same enabled event set and registry-owned JSON schemas;
- the same ChatAgent model, persona prompt, user-visible conversational history, sampling parameters, and output budget, except for the minimum mode-specific event/tool instruction;
- the same normalized internal `EventBatch` and event IDs;
- the same deterministic resolvers, validators, optional low-cost LLM fallback, handlers, timeouts, retries, idempotency keys, concurrency rules, and terminal barriers;
- the same result policies and user-visible behavior, including the two-answer `web_search` flow;
- the same audit event names and metric definitions.

Only these variables change:

| Concern | `dual_agent` | `chat_agent_tools` |
| --- | --- | --- |
| ChatAgent capability context | Compact event names/descriptions | Full enabled tool schemas |
| Event selection | Hidden text event names | Native provider tool calls |
| Initial arguments | Empty or minimal; resolved by shared pipeline | Proposed by ChatAgent tool-call arguments |
| EventAgent LLM fallback | Used only when deterministic resolution is incomplete | Used only when ChatAgent arguments plus deterministic normalization remain incomplete |
| Provider dependency | Requires reliable content streaming | Requires reliable streaming native tool calls and multiple tool calls |

Direct ChatAgent tools do not bypass the shared execution pipeline. ChatAgent proposes calls; the application still validates, authorizes, deduplicates, schedules, executes, and applies result policies.

Provider-required assistant tool-call and tool-result messages are allowed to differ between modes. They are internal protocol overhead, not user-visible conversation, and their prompt/completion tokens must be counted as part of the direct-mode result rather than normalized away.

### 2.2 Expected comparison

Neither mode is assumed to win before measurement:

| Dimension | `dual_agent` | `chat_agent_tools` |
| --- | --- | --- |
| Ordinary-turn LLM calls | One ChatAgent call | One ChatAgent call |
| ChatAgent prompt growth | Concise descriptions per event | Full JSON schema per tool |
| Clear side-effect event | Usually one ChatAgent call plus deterministic execution | Usually one ChatAgent call plus deterministic execution |
| Ambiguous arguments | May add one low-cost EventAgent fallback | ChatAgent arguments may avoid fallback; invalid/incomplete arguments may still add fallback |
| Web search | First ChatAgent answer + search + second ChatAgent answer | Same user-visible flow and minimum calls |
| First-answer behavior | Text protocol explicitly places visible reply before event names | Depends on provider support for visible content alongside streamed tool calls |
| Multiple events | Flat text event list | Multiple native tool calls |
| Capability growth cost | ChatAgent context grows slowly | ChatAgent receives every enabled schema on every turn |
| Provider dependence | Requires reliable content streaming and marker adherence | Requires reliable native tool-call streaming, argument JSON, and multiple calls |

The likely trade-off is that direct tools may reduce argument-fallback calls on event-heavy conversations, while dual-agent mode should reduce prompt tokens and preserve first-answer behavior as the enabled capability set grows.

### 2.3 Generic event kernel and flat built-in plugins

The architecture separates a generic event kernel from concrete built-in event plugins. The plugin registry stays flat: the first version does not introduce event category inheritance, nested capability trees, dynamic installation, or a general workflow engine.

The generic kernel owns only reusable orchestration concepts:

- `EventRequest` and ordered `EventBatch` creation from either model protocol;
- deterministic resolver chains and bounded LLM fallback;
- schema validation and confirmation decisions;
- execution-port invocation;
- deduplication, concurrency, conflicts, cancellation, timeout, and terminal barriers;
- normalized `EventResult` values and result-policy aggregation;
- audit and performance spans.

The kernel must not import, compare, or branch on built-in event names.

A built-in plugin owns one concrete capability:

- its namespaced event name and schema version;
- the concise ChatAgent catalog description and provider tool schema;
- deterministic resolver and validator;
- optional LLM fallback policy;
- execution handler/port adapter;
- confirmation, risk, idempotency, concurrency, and result policies;
- focused contract tests.

Initial plugin names are:

- `session.terminate`;
- `device.volume.adjust`;
- `calendar.schedule.create`;
- `knowledge.web.search`.

Adding a capability should normally require only:

1. one new built-in plugin definition containing its namespaced name, schema version, concise ChatAgent description, argument schema, resolution strategy, result policy, timeout, concurrency metadata, and terminal flag;
2. one deterministic resolver/validator, with optional LLM fallback configuration;
3. one handler or service-owned port adapter;
4. focused contract, execution, failure, and concurrency tests.

The parsers, ChatAgent protocols, Runtime, and generic kernel must not require an event-name-specific branch. They operate on the metadata declared by each plugin definition. The event-specific flow sections below describe built-in plugin behavior and acceptance criteria, not hardcoded Runtime architecture.

Multiple events remain a flat ordered list in one batch. Generic batch orchestration handles deduplication, conflict metadata, bounded concurrency, result aggregation, and terminal barriers.

The flat plugin registry should evolve only when real additions reveal repeated structures that cannot be expressed cleanly with definition metadata. Examples include several events sharing one authorization flow, a multi-step transaction/rollback requirement, or recurring dependencies between event outputs. Until then, those abstractions are deliberately out of scope.

## 3. Event Flows

### 3.1 Terminate session

1. The user expresses a clear intention to leave or end the voice conversation.
2. ChatAgent immediately streams a short, natural farewell.
3. ChatAgent requests `terminate_session` through the configured text-event or native tool-call protocol.
4. EventAgent executes a dialogue-session termination command.
5. On success, the runtime marks the session terminal and suppresses another ChatAgent round.
6. On failure, the runtime records the failure and may emit one concise corrective message instead of pretending that the session ended.

This event should terminate the dialogue through the dialogue/session boundary. It should not be implemented as a direct MQTT action.

### 3.2 Adjust device volume

1. The user asks for an absolute volume, relative change, mute, or unmute.
2. ChatAgent immediately acknowledges the requested action naturally.
3. ChatAgent requests `adjust_device_volume` through the configured protocol.
4. EventAgent resolves normalized arguments such as:
   - `mode`: `absolute`, `relative`, `mute`, or `unmute`;
   - `value`: an absolute target when present;
   - `delta`: a signed relative change when present.
5. The handler sends a device-action command through the device-action/dispatch boundary.
6. A successful action does not produce a second spoken answer. A failed or rejected action produces a short corrective follow-up.

Validation must clamp or reject values outside the supported range and must distinguish "set to 30" from "turn it up by 30".

### 3.3 Add schedule

1. The user asks the toy to remember a future activity, reminder, or calendar item.
2. ChatAgent immediately acknowledges the request naturally and requests `add_schedule` through the configured protocol.
3. EventAgent first attempts deterministic extraction and normalization of:
   - title or activity;
   - local date and time;
   - timezone;
   - optional recurrence;
   - optional reminder offset.
4. If required fields cannot be resolved deterministically from the utterance and recent context, EventAgent uses its LLM fallback with only the schedule schema and minimal relevant history.
5. If the time remains ambiguous after fallback, no schedule is created and ChatAgent asks one focused clarification question.
6. On success, the runtime returns a deterministic confirmation containing the normalized date, time, and title. This confirmation does not require a second ChatAgent LLM call.

Schedule execution must use an idempotency key so retransmission or duplicate event detection does not create duplicate entries. Relative phrases such as "tomorrow morning" are resolved against the session timezone and a captured request timestamp.

### 3.4 Web search with two answers

1. ChatAgent immediately gives a useful preliminary response in a natural tone.
2. When freshness or external facts matter, ChatAgent mentions that it will check and requests `web_search` through the configured protocol after the visible reply where the provider supports content plus tool calls.
3. EventAgent derives a focused search query from the current user request and relevant conversation history.
4. The search adapter returns structured results, including query, answer evidence, source title, source location, and retrieval time where available.
5. The runtime sends a compact search-result summary back to ChatAgent.
6. ChatAgent produces a second answer that:
   - clearly sounds like an update after checking;
   - answers with the retrieved evidence;
   - corrects the preliminary answer if necessary;
   - avoids repeating the entire first response;
   - preserves source information for clients that can display it.

The first answer must not fabricate current facts. When the answer depends heavily on search, it should provide a natural bridge or a clearly qualified preliminary view rather than guessing.

### 3.5 Multiple events in one response

ChatAgent may output multiple event names in the hidden event block or multiple native tool calls in one response. Both protocols produce the same flat ordered event batch. The shared pipeline resolves and executes independent events concurrently with a bounded concurrency limit.

Examples that may run concurrently:

- adjust volume and add a schedule;
- add a schedule and search related information;
- adjust volume and search the web.

Conflict policy:

- duplicate event names are deduplicated only when their normalized arguments are equivalent;
- read-only events such as `web_search` may run alongside side-effecting events;
- `terminate_session` is dispatched after other accepted events have reached a terminal result, with a short overall deadline, so ending the dialogue does not cancel an already acknowledged schedule or device action;
- an event failure does not cancel unrelated events;
- follow-up outputs are aggregated by policy: deterministic confirmations first, then at most one ChatAgent LLM follow-up for all results that require natural synthesis.

## 4. Generic Kernel, Runtime, and Plugin Registry

The design extracts a generic kernel from Runtime and extends registry-owned plugin definitions with execution metadata rather than adding event-name conditionals throughout orchestration.

Conceptually, a built-in plugin definition contains:

- name, description, and JSON argument schema;
- argument-generation behavior;
- deterministic parser/normalizer and required-field rules;
- whether LLM fallback is allowed;
- execution handler;
- result policy;
- timeout and retry policy;
- concurrency class and conflict metadata;
- whether failure is safe to expose to ChatAgent;
- whether the event makes the session terminal.

Runtime remains responsible for session/turn and model-stream lifecycle. It delegates event work to the generic kernel:

1. stream ChatAgent output;
2. parse hidden text events or collect native tool calls into the same internal event batch;
3. submit the flat batch to the event kernel;
4. let the kernel resolve, validate, confirm, deduplicate, schedule, execute, and normalize results using plugin metadata;
5. publish normalized audit/tool results;
6. apply the kernel's aggregated continuation decision;
7. optionally run one aggregated follow-up ChatAgent round.

EventAgent supplies optional LLM fallback to the kernel; it does not own built-in event behavior. Production side effects remain behind service-owned ports/adapters implemented by plugins.

The local fast Gate for obvious high-confidence intents remains a later optimization. It should be evaluated only after the two configured model-driven modes establish a baseline and production-like measurements show that event-catalog/tool-schema cost or event-selection errors justify the added routing complexity.

## 5. Context and Token Boundaries

ChatAgent should receive:

- the user-visible conversation;
- its reply/personality instructions;
- in `dual_agent` mode, a compact event catalog containing only enabled event names and concise descriptions;
- in `chat_agent_tools` mode, the enabled schemas produced by the same event registry.

In `dual_agent` mode, ChatAgent should not receive:

- provider tool schemas;
- tool execution details;
- raw search documents;
- internal service configuration;
- prior tool-role protocol messages.

EventAgent should receive:

- the selected event schema only;
- the smallest relevant slice of user/assistant conversation history;
- EventAgent-specific instructions and runtime metadata needed for execution.

For `web_search`, only the compact result summary enters the second ChatAgent call. Raw provider payloads stay in audit/debug data.

For deterministic resolution, no EventAgent LLM request is made. When fallback is needed, EventAgent should use a lower-cost model and a small token limit because its output is structured arguments rather than conversational text.

## 6. Error Handling and Safety

- Event execution is bounded by per-event timeouts.
- Termination, volume, and schedule events are not automatically retried when duplicate side effects are possible; handlers should support an idempotency key based on session, turn, event ID, and normalized arguments.
- Web search may retry transient read-only failures once within its latency budget.
- Invalid or incomplete volume requests are returned as structured failures so ChatAgent can ask one focused clarification question.
- Ambiguous schedule time is never guessed into a side effect; it produces a focused clarification request.
- Parallel event batches use per-event timeouts plus one batch deadline; timed-out events do not discard completed sibling results.
- EventAgent failure must not corrupt conversation history or block future user turns.
- Audit data records event detection, deterministic-resolution outcome, whether LLM fallback was used, execution result, policy decision, timings, and redacted raw provider data.

## 7. Service Integration Boundaries

`agent-lab` is the behavior and protocol proving ground. Production integration belongs to `mituan-service` and must use its existing boundaries:

- session termination through the dialogue/session service boundary;
- volume adjustment through the device-action dispatcher and published device signal, not direct intent-to-MQTT coupling;
- schedule creation through a calendar/reminder application port with an explicit user/timezone contract;
- web search through an external-capability adapter that returns a stable internal search-result contract.

The current service voice reply path does not yet wire web search into its model request, so real search integration is a separate service-repository change.

Because implementation spans repositories, implementation planning must be split:

- one `agent-lab` plan for runtime policies, mock/contract tools, UI/audit visibility, and tests;
- one `mituan-service` plan for production ports, adapters, device/session commands, configuration, and integration tests.

Each repository plan must load that repository's `AGENTS.md` and maintain its own `todos.md` execution order and status.

## 8. Verification

### Agent Lab behavior

- `dual_agent` sends `tools=[]`, injects the compact event catalog, and extracts hidden text events.
- `chat_agent_tools` sends exactly the enabled registry schemas and converts native tool calls into the same internal event batch.
- Switching mode does not change enabled events, handlers, normalized arguments, result policies, concurrency, or user-visible success/failure semantics.
- Both modes support multiple events in one model response and execute the resulting batch through the same bounded parallel orchestrator.
- The event catalog and full tool-schema prompt costs are measured independently.
- Both modes are prompted to emit the natural first response before event execution; if a native tool provider suppresses visible content until after tool execution, that is recorded as a direct-mode behavior/consistency failure rather than hidden by the harness.
- Hidden event blocks never appear in user-visible deltas.
- Deterministic argument resolution bypasses the EventAgent LLM, while incomplete or ambiguous cases use the configured fallback.
- `terminate_session` yields farewell -> accepted sibling events finish or time out -> successful termination -> no second ChatAgent call.
- `adjust_device_volume` yields acknowledgement -> action result, with no second answer on success and a corrective answer on failure.
- `add_schedule` yields acknowledgement -> normalized execution -> deterministic confirmation, without a second LLM call.
- `web_search` yields first answer -> search execution -> distinct second answer grounded in the returned result.
- Multiple independent events execute concurrently, preserve input order in reported results, and isolate failures.
- A mixed batch produces at most one extra ChatAgent LLM call even when several events require natural follow-up.
- Search failure still yields a natural, non-blocking second message.
- Event IDs make duplicate execution idempotent.

### Service integration

- Termination cleans up active dialogue state across normal, stop, disconnect, and transport-error paths.
- Volume commands validate absolute/relative semantics and reach the device through the established dispatcher/protocol boundary.
- Schedule commands resolve relative time against the correct timezone and do not create duplicates.
- Search configuration is environment-specific and contains no committed credentials.
- End-to-end tracing can correlate one user turn, one or two ChatAgent responses, parallel EventAgent work, and downstream side effects.

### Performance acceptance

- Replay the same conversation/event dataset in `dual_agent` and `chat_agent_tools` modes with the same model parameters and enabled events.
- Compare each mode with all events disabled and with all four enabled.
- The first answer must not wait for search completion.
- Report first-answer TTFT, first-answer completion time, ChatAgent prompt/completion tokens, LLM calls per turn, deterministic-resolution hit rate, EventAgent fallback rate/tokens, event-dispatch delay, per-event latency, second-answer TTFT, total turn latency, and estimated cost separately.
- Report event-selection precision/recall, multiple-event recall, valid-argument rate before and after normalization, execution success rate, and first-answer quality for each mode.
- Track average LLM calls per turn and cost per successful event; the target is one ChatAgent call for ordinary turns, zero EventAgent LLM calls for deterministic events, and a second ChatAgent call only when result policy requires natural synthesis.

Expected hypotheses, not predetermined conclusions:

- `dual_agent` should use fewer ChatAgent prompt tokens and scale better as tool schemas grow;
- `chat_agent_tools` may avoid EventAgent fallback calls when ChatAgent produces complete arguments;
- provider-native tool calls may delay or suppress visible content before execution, so actual TTFT must be measured rather than assumed;
- the better production default depends on event frequency, schema size, provider behavior, argument accuracy, and total cost per successful turn.

## 9. Non-Goals

- Replacing the current compatible Chat Completions client with a different API solely for web search.
- Moving device protocol or MQTT details into EventAgent.
- Making every event produce a second answer.
- Implementing the local fast Gate in the first iteration.
- Introducing event inheritance, a nested capability taxonomy, a plugin lifecycle, or a general DAG/workflow engine before repeated requirements justify them.
- Broad refactoring of the service voice pipeline outside the boundaries required by these four events.