ChatAgent Event Selection Prompt Design

Date: 2026-07-15 Status: approved for implementation planning

Context

In dual_agent mode, ChatAgent selects EventAgent capabilities by appending event names inside an <agent_events> block after its visible reply. The current generated prompt explains the catalog and output format, but treats event selection as optional and does not define when an event is required.

Recent local audit evidence shows the practical failure mode: ChatAgent says it will adjust volume or check weather, but emits no event. Examples include 声音调到90, 调大一点音量, and later weather requests. The model therefore produces a plausible conversational promise without causing the real action.

The generated prompt currently exists in two equivalent construction paths:

ToolRegistry.chat_event_system_message() supplies the runtime fallback for API clients that did not submit an EventPrompt.
buildAvailableEventsPrompt() generates the read-only EventPrompt shown and submitted by the browser console.

Both paths must retain the same behavioral contract. The text-event parser and the <agent_events> wire format are already stable and are not the source of the observed missed triggers.

Goals

Balance event recall against false positives.
Require an event whenever ChatAgent promises or claims a supported external action.
Trigger low-risk, read-only, or reversible capabilities from clear natural intent, even when the user does not phrase it as an imperative.
Ask one concise clarification before ambiguous or incomplete higher-impact actions.
Keep ordinary conversation, hypothetical discussion, unsupported requests, and capability questions event-free.
Preserve the existing event block protocol, parser, EventAgent argument resolution, event kernel, and invocation-mode boundaries.
Continue recognizing old generated EventPrompts when restoring sessions or workspace snapshots.

Non-goals

No separate intent-classification model call.
No provider-native tool schema changes.
No EventAgent argument protocol changes; ChatAgent still emits names only.
No event-kernel, handler, session, WebSocket, or active-turn behavior change.
No guarantee that every stochastic model follows the prompt perfectly; live evaluation measures adherence but does not replace deterministic contracts.

Approaches Considered

1. Decision-oriented prompt only

Replace the weak optional wording with explicit selection rules, consistency invariants, and compact positive and negative examples. Keep the current protocol and model-call count.

This is the selected approach because it directly addresses the observed failure, has the smallest runtime scope, and can be evaluated independently before introducing new architecture.

2. Metadata-driven trigger policies

Expose trigger and confirmation metadata through the event catalog and render event-specific policy dynamically. This scales better for a large heterogeneous registry, but requires domain, catalog API, frontend, and compatibility changes. It remains a possible follow-up if prompt-only evaluation shows that generic rules are insufficient.

3. Separate event-intent classifier

Run a dedicated classification step before ChatAgent response generation. This could improve isolation but adds latency, token cost, another failure mode, and coordination complexity. It is not justified before measuring the prompt-only approach.

Selected Prompt Contract

The generated EventPrompt will identify itself as protocol version 2 and use imperative rules. Its dynamic catalog remains name plus description only.

The prompt must communicate these rules:

Decide whether the user's latest intent requires any available event.
When a supported real-world action or fresh external-data request is clear, emit the matching event.
If the visible reply says or implies that ChatAgent will perform, has performed, is checking, is changing, is scheduling, or is ending something, the same reply must append the corresponding event. ChatAgent must never promise a supported action without emitting its event.
Low-risk, read-only, and reversible actions may be inferred from clear natural intent. 声音太小了 can select device.volume.adjust, and 明天天气怎么样 can select knowledge.web.search.
For ambiguous or incomplete higher-impact actions, ask one concise clarification and emit no event yet. A complete and unambiguous request does not require redundant confirmation.
Do not emit events for ordinary conversation, hypothetical statements, discussion about a capability, unsupported behavior, or a request that does not match an available event.
Multiple independent clear intents may emit multiple exact event names, one per line.
The event block is machine-control metadata, not part of the visible reply. It is allowed even when another prompt prohibits Markdown, lists, JSON, or other visible formatting.
When no event is selected, omit the event block entirely.
Never include parameters, explanations, bullets, code fences, or invented names inside the block.

The generated shape will remain:

Event selection protocol v2.
Available events:
- event.name: description
...
[decision rules and compact examples]
First write the user-facing reply normally. If one or more events are selected,
append exactly this block after the visible reply:
<agent_events>
event_name
</agent_events>
Use exact available event names only, one per line. Do not include parameters.

Decision Examples

The prompt will include a small, generic set of examples that match the current production-shaped capabilities without teaching argument extraction:

User intent	Expected selection behavior
`声音太小了`	Reply naturally and emit `device.volume.adjust`.
`声音调到90`	Reply naturally and emit `device.volume.adjust`.
`明天天气怎么样`	Reply naturally and emit `knowledge.web.search`.
`你回复太慢了`	No event; response latency is not device volume.
`你能调音量吗`	No event; this asks about capability rather than requesting action.
`提醒我开会`	Ask for missing scheduling details and emit no event.
`我明天下午三点要开会`	No event; this is a statement unless the user requests scheduling.
A complete request to create a schedule	Emit `calendar.schedule.create`.
An explicit goodbye or session-ending request	Emit `session.terminate`.
Ordinary greeting or emotional conversation	No event.

Examples are selection guidance only. EventAgent remains responsible for arguments and validation.

Data Flow and Compatibility

The browser continues to generate a read-only EventPrompt from selected tools. In dual_agent, exactly one managed EventPrompt is serialized. In chat_agent_tools, generated EventPrompt content remains excluded and provider tool schemas remain the only capability-selection surface.

The runtime continues to inject its registry-generated prompt only when no event instructions are already present. The Python and JavaScript prompt builders will use the same version marker, rule ordering, examples, catalog format, and output suffix.

isGeneratedEventPromptContent() must recognize both:

the legacy prefix You may request EventAgent work using this text protocol.;
the new Event selection protocol v2. prefix.

Recognition remains conservative: a known prefix alone is insufficient. The content must also contain the event block markers and the exact-name/no- parameters instruction. This preserves cleanup of historical generated prompts without deleting ordinary custom prompts that merely mention event syntax.

Verification Design

Implementation will use TDD.

Deterministic tests will first fail against the current prompt and then prove:

the backend prompt contains the version marker, mandatory action/event consistency rule, low-risk inference rule, higher-impact clarification rule, negative boundaries, exact output block, and compact examples;
the frontend prompt contains the same behavioral anchors and catalog output;
old and new generated EventPrompt signatures are both recognized;
ordinary custom prompts containing partial markers remain retained;
payload singleton and invocation-mode filtering behavior remain unchanged;
runtime fallback injection still supplies exactly one catalog and no provider tools in dual_agent;
parser behavior is unchanged.

Focused verification commands will cover the prompt/runtime tests and frontend source/behavior tests, followed by JavaScript syntax, the full suite, app import, and diff checks.

If a local live provider target and API key are available, run a fresh A/B evaluation with the same model and parameters. At minimum, evaluate the ten decision examples above across old and new prompts and record both missed required events and false positives. Existing historical SQLite sessions are evidence for the problem but must not be reused as an A/B input source because their accumulated context and prompt order differ.

Risks and Controls

Stronger rules may increase false positives. Negative examples and the capability-question/hypothetical boundaries control this risk.
Generic risk language may not scale to future events. If live evaluation exposes event-specific ambiguity, follow up with metadata-driven policy rather than continually expanding a monolithic prompt.
Prompt duplication between Python and JavaScript can drift. Tests assert the shared behavioral anchors and exact protocol suffix in both paths.
Historical snapshots may contain the old generated prompt. Dual-signature detection preserves deduplication and outbound singleton guarantees.
Other system prompts may prohibit visible formatting. The v2 prompt explicitly defines the event block as hidden machine metadata so those instructions do not suppress event emission.

Success Criteria

Direct or naturally expressed volume requests and fresh-data questions cause the matching event instead of an unsupported verbal promise.
Response-latency complaints, capability questions, ordinary chat, and incomplete scheduling requests do not produce unrelated events.
No protocol, parser, provider-tool mode, EventAgent argument, or event-kernel regression occurs.
Old and new EventPrompt snapshots remain safely deduplicated.
Deterministic verification passes, and any available live A/B evidence records both recall and false-positive outcomes rather than reporting only successful triggers.

2026-07-15-chat-agent-event-prompt-design.md 10 KB Historie Surový