agent-lab/docs/plans/2026-07-12-chat-event-agent-production-events-design.md

ChatAgent / EventAgent Production Events Design

Status: approved design

Date: 2026-07-12

1. Goal

Evaluate two configurable capability-invocation architectures for the service voice-reply path, primarily targeting a low-latency conversational toy:

• ChatAgent owns low-latency, natural, user-facing speech.
• the shared event pipeline owns extensible argument resolution, validation, concurrency, execution, and result policy;
• in dual-agent mode ChatAgent selects event names and EventAgent supplies missing-argument fallback, while direct mode lets ChatAgent select events and propose arguments through native tool calls.

The first production events are:

• terminate the current voice session;
• adjust device volume;
• add a calendar/reminder schedule;
• search the web and provide a second, search-grounded answer.

The primary success criterion is to measure whether keeping provider tool schemas off ChatAgent's first-token critical path actually improves reply latency and cost without reducing event accuracy. The lab must support both direct ChatAgent tool calling and the dual-agent text-event design under one consistent capability contract.

2. Architecture Decision

Keep one shared flat event registry and make the event-selection path configurable. The two experimental modes differ only in how events are selected and how initial arguments are proposed; they share argument normalization, execution, concurrency, result policies, and user-visible behavior.

Configuration uses one explicit tool_invocation_mode value:

• dual_agent: ChatAgent receives a compact event catalog but no provider tools. It streams its visible reply and appends internal event names in the text-event block. EventAgent resolves missing arguments.
• chat_agent_tools: ChatAgent receives the enabled provider tool schemas and emits native tool calls, including initial arguments. The shared event pipeline validates and normalizes those arguments before execution.

In dual_agent mode, the parser hides the event block from the user and dispatches the named events to EventAgent. In chat_agent_tools mode, the streaming client collects native tool-call deltas while continuing to expose any visible assistant content allowed by the provider.

Both modes convert model output into the same internal event batch. The shared pipeline performs deterministic parsing and normalization first, then a small LLM fallback only when required fields remain ambiguous or missing. It validates the resolved arguments and invokes the same handlers through the registry.

ChatAgent may emit multiple event names in one response. Independent events are resolved and executed concurrently. The runtime applies an explicit conflict policy before execution so terminal or mutually exclusive side effects do not race unpredictably.

Each event definition declares a result policy so the runtime does not hardcode behavior by event name:

Result policy	Runtime behavior
terminate	Execute after ChatAgent's farewell, end the session on success, and do not run ChatAgent again.
silent_success	Keep ChatAgent's first acknowledgement as the normal response; run a corrective follow-up only if execution fails.
template_follow_up	Produce a deterministic confirmation from the normalized execution result without another LLM call.
llm_follow_up	Feed a compact structured result back to ChatAgent and produce a second user-facing answer.

Initial mapping:

Event	Result policy
terminate_session	terminate
adjust_device_volume	silent_success
add_schedule	template_follow_up
web_search	llm_follow_up

2.1 Experimental consistency contract

The comparison must not become two separate implementations. Both modes use:

• the same enabled event set and registry-owned JSON schemas;
• the same ChatAgent model, persona prompt, user-visible conversational history, sampling parameters, and output budget, except for the minimum mode-specific event/tool instruction;
• the same normalized internal EventBatch and event IDs;
• the same deterministic resolvers, validators, optional low-cost LLM fallback, handlers, timeouts, retries, idempotency keys, concurrency rules, and terminal barriers;
• the same result policies and user-visible behavior, including the two-answer web_search flow;
• the same audit event names and metric definitions.

Only these variables change:

Concern	dual_agent	chat_agent_tools
ChatAgent capability context	Compact event names/descriptions	Full enabled tool schemas
Event selection	Hidden text event names	Native provider tool calls
Initial arguments	Empty or minimal; resolved by shared pipeline	Proposed by ChatAgent tool-call arguments
EventAgent LLM fallback	Used only when deterministic resolution is incomplete	Used only when ChatAgent arguments plus deterministic normalization remain incomplete
Provider dependency	Requires reliable content streaming	Requires reliable streaming native tool calls and multiple tool calls

Direct ChatAgent tools do not bypass the shared execution pipeline. ChatAgent proposes calls; the application still validates, authorizes, deduplicates, schedules, executes, and applies result policies.

Provider-required assistant tool-call and tool-result messages are allowed to differ between modes. They are internal protocol overhead, not user-visible conversation, and their prompt/completion tokens must be counted as part of the direct-mode result rather than normalized away.

2.2 Expected comparison

Neither mode is assumed to win before measurement:

Dimension	dual_agent	chat_agent_tools
Ordinary-turn LLM calls	One ChatAgent call	One ChatAgent call
ChatAgent prompt growth	Concise descriptions per event	Full JSON schema per tool
Clear side-effect event	Usually one ChatAgent call plus deterministic execution	Usually one ChatAgent call plus deterministic execution
Ambiguous arguments	May add one low-cost EventAgent fallback	ChatAgent arguments may avoid fallback; invalid/incomplete arguments may still add fallback
Web search	First ChatAgent answer + search + second ChatAgent answer	Same user-visible flow and minimum calls
First-answer behavior	Text protocol explicitly places visible reply before event names	Depends on provider support for visible content alongside streamed tool calls
Multiple events	Flat text event list	Multiple native tool calls
Capability growth cost	ChatAgent context grows slowly	ChatAgent receives every enabled schema on every turn
Provider dependence	Requires reliable content streaming and marker adherence	Requires reliable native tool-call streaming, argument JSON, and multiple calls

The likely trade-off is that direct tools may reduce argument-fallback calls on event-heavy conversations, while dual-agent mode should reduce prompt tokens and preserve first-answer behavior as the enabled capability set grows.

2.3 Flat extensibility model

The initial event model stays flat. There is one registry of self-contained event definitions; the first version does not introduce event category inheritance, nested capability trees, plugins, or a general workflow engine.

Adding a capability should normally require only:

1. one new event definition containing its name, concise ChatAgent description, argument schema, resolution strategy, result policy, timeout, concurrency metadata, and terminal flag;
2. one deterministic resolver/validator, with optional LLM fallback configuration;
3. one handler or service-owned port adapter;
4. focused contract, execution, failure, and concurrency tests.

The parsers, ChatAgent protocols, Runtime, and shared event orchestration must not require an event-name-specific branch. They operate on the metadata declared by each definition. The event-specific flow sections below describe initial behavior and acceptance criteria, not hardcoded Runtime architecture.

Multiple events remain a flat ordered list in one batch. Generic batch orchestration handles deduplication, conflict metadata, bounded concurrency, result aggregation, and terminal barriers.

The flat model should evolve only when real additions reveal repeated structures that cannot be expressed cleanly with definition metadata. Examples include several events sharing one authorization flow, a multi-step transaction/rollback requirement, or recurring dependencies between event outputs. Until then, those abstractions are deliberately out of scope.

3. Event Flows

3.1 Terminate session

1. The user expresses a clear intention to leave or end the voice conversation.
2. ChatAgent immediately streams a short, natural farewell.
3. ChatAgent requests terminate_session through the configured text-event or native tool-call protocol.
4. EventAgent executes a dialogue-session termination command.
5. On success, the runtime marks the session terminal and suppresses another ChatAgent round.
6. On failure, the runtime records the failure and may emit one concise corrective message instead of pretending that the session ended.

This event should terminate the dialogue through the dialogue/session boundary. It should not be implemented as a direct MQTT action.

3.2 Adjust device volume

1. The user asks for an absolute volume, relative change, mute, or unmute.
2. ChatAgent immediately acknowledges the requested action naturally.
3. ChatAgent requests adjust_device_volume through the configured protocol.
4. EventAgent resolves normalized arguments such as:
   • mode: absolute, relative, mute, or unmute;
   • value: an absolute target when present;
   • delta: a signed relative change when present.
5. The handler sends a device-action command through the device-action/dispatch boundary.
6. A successful action does not produce a second spoken answer. A failed or rejected action produces a short corrective follow-up.

Validation must clamp or reject values outside the supported range and must distinguish "set to 30" from "turn it up by 30".

3.3 Add schedule

1. The user asks the toy to remember a future activity, reminder, or calendar item.
2. ChatAgent immediately acknowledges the request naturally and requests add_schedule through the configured protocol.
3. EventAgent first attempts deterministic extraction and normalization of:
   • title or activity;
   • local date and time;
   • timezone;
   • optional recurrence;
   • optional reminder offset.
4. If required fields cannot be resolved deterministically from the utterance and recent context, EventAgent uses its LLM fallback with only the schedule schema and minimal relevant history.
5. If the time remains ambiguous after fallback, no schedule is created and ChatAgent asks one focused clarification question.
6. On success, the runtime returns a deterministic confirmation containing the normalized date, time, and title. This confirmation does not require a second ChatAgent LLM call.

Schedule execution must use an idempotency key so retransmission or duplicate event detection does not create duplicate entries. Relative phrases such as "tomorrow morning" are resolved against the session timezone and a captured request timestamp.

3.4 Web search with two answers

1. ChatAgent immediately gives a useful preliminary response in a natural tone.
2. When freshness or external facts matter, ChatAgent mentions that it will check and requests web_search through the configured protocol after the visible reply where the provider supports content plus tool calls.
3. EventAgent derives a focused search query from the current user request and relevant conversation history.
4. The search adapter returns structured results, including query, answer evidence, source title, source location, and retrieval time where available.
5. The runtime sends a compact search-result summary back to ChatAgent.
6. ChatAgent produces a second answer that:
   • clearly sounds like an update after checking;
   • answers with the retrieved evidence;
   • corrects the preliminary answer if necessary;
   • avoids repeating the entire first response;
   • preserves source information for clients that can display it.

The first answer must not fabricate current facts. When the answer depends heavily on search, it should provide a natural bridge or a clearly qualified preliminary view rather than guessing.

3.5 Multiple events in one response

ChatAgent may output multiple event names in the hidden event block or multiple native tool calls in one response. Both protocols produce the same flat ordered event batch. The shared pipeline resolves and executes independent events concurrently with a bounded concurrency limit.

Examples that may run concurrently:

• adjust volume and add a schedule;
• add a schedule and search related information;
• adjust volume and search the web.

Conflict policy:

• duplicate event names are deduplicated only when their normalized arguments are equivalent;
• read-only events such as web_search may run alongside side-effecting events;
• terminate_session is dispatched after other accepted events have reached a terminal result, with a short overall deadline, so ending the dialogue does not cancel an already acknowledged schedule or device action;
• an event failure does not cancel unrelated events;
• follow-up outputs are aggregated by policy: deterministic confirmations first, then at most one ChatAgent LLM follow-up for all results that require natural synthesis.

4. Runtime and Registry Changes

The design extends the registry-owned event definition with execution metadata rather than adding event-name conditionals throughout the runtime.

Conceptually, an event definition contains:

• name, description, and JSON argument schema;
• argument-generation behavior;
• deterministic parser/normalizer and required-field rules;
• whether LLM fallback is allowed;
• execution handler;
• result policy;
• timeout and retry policy;
• concurrency class and conflict metadata;
• whether failure is safe to expose to ChatAgent;
• whether the event makes the session terminal.

The runtime remains responsible for orchestration and ordering:

1. stream ChatAgent output;
2. parse hidden text events or collect native tool calls into the same internal event batch;
3. resolve event arguments using deterministic parsers first and LLM fallback only when needed;
4. apply deduplication, conflict, and concurrency rules;
5. dispatch independent EventAgent work concurrently;
6. publish audit/tool results;
7. apply each event's result policy;
8. optionally run one aggregated follow-up ChatAgent round.

EventAgent remains responsible for argument resolution, validation, and handler invocation. Production side effects remain behind service-owned ports/adapters.

The local fast Gate for obvious high-confidence intents remains a later optimization. It should be evaluated only after the two configured model-driven modes establish a baseline and production-like measurements show that event-catalog/tool-schema cost or event-selection errors justify the added routing complexity.

5. Context and Token Boundaries

ChatAgent should receive:

• the user-visible conversation;
• its reply/personality instructions;
• in dual_agent mode, a compact event catalog containing only enabled event names and concise descriptions;
• in chat_agent_tools mode, the enabled schemas produced by the same event registry.

In dual_agent mode, ChatAgent should not receive:

• provider tool schemas;
• tool execution details;
• raw search documents;
• internal service configuration;
• prior tool-role protocol messages.

EventAgent should receive:

• the selected event schema only;
• the smallest relevant slice of user/assistant conversation history;
• EventAgent-specific instructions and runtime metadata needed for execution.

For web_search, only the compact result summary enters the second ChatAgent call. Raw provider payloads stay in audit/debug data.

For deterministic resolution, no EventAgent LLM request is made. When fallback is needed, EventAgent should use a lower-cost model and a small token limit because its output is structured arguments rather than conversational text.

6. Error Handling and Safety

• Event execution is bounded by per-event timeouts.
• Termination, volume, and schedule events are not automatically retried when duplicate side effects are possible; handlers should support an idempotency key based on session, turn, event ID, and normalized arguments.
• Web search may retry transient read-only failures once within its latency budget.
• Invalid or incomplete volume requests are returned as structured failures so ChatAgent can ask one focused clarification question.
• Ambiguous schedule time is never guessed into a side effect; it produces a focused clarification request.
• Parallel event batches use per-event timeouts plus one batch deadline; timed-out events do not discard completed sibling results.
• EventAgent failure must not corrupt conversation history or block future user turns.
• Audit data records event detection, deterministic-resolution outcome, whether LLM fallback was used, execution result, policy decision, timings, and redacted raw provider data.

7. Service Integration Boundaries

agent-lab is the behavior and protocol proving ground. Production integration belongs to mituan-service and must use its existing boundaries:

• session termination through the dialogue/session service boundary;
• volume adjustment through the device-action dispatcher and published device signal, not direct intent-to-MQTT coupling;
• schedule creation through a calendar/reminder application port with an explicit user/timezone contract;
• web search through an external-capability adapter that returns a stable internal search-result contract.

The current service voice reply path does not yet wire web search into its model request, so real search integration is a separate service-repository change.

Because implementation spans repositories, implementation planning must be split:

• one agent-lab plan for runtime policies, mock/contract tools, UI/audit visibility, and tests;
• one mituan-service plan for production ports, adapters, device/session commands, configuration, and integration tests.

Each repository plan must load that repository's AGENTS.md and maintain its own todos.md execution order and status.

8. Verification

Agent Lab behavior

• dual_agent sends tools=[], injects the compact event catalog, and extracts hidden text events.
• chat_agent_tools sends exactly the enabled registry schemas and converts native tool calls into the same internal event batch.
• Switching mode does not change enabled events, handlers, normalized arguments, result policies, concurrency, or user-visible success/failure semantics.
• Both modes support multiple events in one model response and execute the resulting batch through the same bounded parallel orchestrator.
• The event catalog and full tool-schema prompt costs are measured independently.
• Both modes are prompted to emit the natural first response before event execution; if a native tool provider suppresses visible content until after tool execution, that is recorded as a direct-mode behavior/consistency failure rather than hidden by the harness.
• Hidden event blocks never appear in user-visible deltas.
• Deterministic argument resolution bypasses the EventAgent LLM, while incomplete or ambiguous cases use the configured fallback.
• terminate_session yields farewell -> accepted sibling events finish or time out -> successful termination -> no second ChatAgent call.
• adjust_device_volume yields acknowledgement -> action result, with no second answer on success and a corrective answer on failure.
• add_schedule yields acknowledgement -> normalized execution -> deterministic confirmation, without a second LLM call.
• web_search yields first answer -> search execution -> distinct second answer grounded in the returned result.
• Multiple independent events execute concurrently, preserve input order in reported results, and isolate failures.
• A mixed batch produces at most one extra ChatAgent LLM call even when several events require natural follow-up.
• Search failure still yields a natural, non-blocking second message.
• Event IDs make duplicate execution idempotent.

Service integration

• Termination cleans up active dialogue state across normal, stop, disconnect, and transport-error paths.
• Volume commands validate absolute/relative semantics and reach the device through the established dispatcher/protocol boundary.
• Schedule commands resolve relative time against the correct timezone and do not create duplicates.
• Search configuration is environment-specific and contains no committed credentials.
• End-to-end tracing can correlate one user turn, one or two ChatAgent responses, parallel EventAgent work, and downstream side effects.

Performance acceptance

• Replay the same conversation/event dataset in dual_agent and chat_agent_tools modes with the same model parameters and enabled events.
• Compare each mode with all events disabled and with all four enabled.
• The first answer must not wait for search completion.
• Report first-answer TTFT, first-answer completion time, ChatAgent prompt/completion tokens, LLM calls per turn, deterministic-resolution hit rate, EventAgent fallback rate/tokens, event-dispatch delay, per-event latency, second-answer TTFT, total turn latency, and estimated cost separately.
• Report event-selection precision/recall, multiple-event recall, valid-argument rate before and after normalization, execution success rate, and first-answer quality for each mode.
• Track average LLM calls per turn and cost per successful event; the target is one ChatAgent call for ordinary turns, zero EventAgent LLM calls for deterministic events, and a second ChatAgent call only when result policy requires natural synthesis.

Expected hypotheses, not predetermined conclusions:

• dual_agent should use fewer ChatAgent prompt tokens and scale better as tool schemas grow;
• chat_agent_tools may avoid EventAgent fallback calls when ChatAgent produces complete arguments;
• provider-native tool calls may delay or suppress visible content before execution, so actual TTFT must be measured rather than assumed;
• the better production default depends on event frequency, schema size, provider behavior, argument accuracy, and total cost per successful turn.

9. Non-Goals

• Replacing the current compatible Chat Completions client with a different API solely for web search.
• Moving device protocol or MQTT details into EventAgent.
• Making every event produce a second answer.
• Implementing the local fast Gate in the first iteration.
• Introducing event inheritance, a nested capability taxonomy, a plugin lifecycle, or a general DAG/workflow engine before repeated requirements justify them.
• Broad refactoring of the service voice pipeline outside the boundaries required by these four events.