|
|
@@ -0,0 +1,226 @@
|
|
|
+# ChatAgent Event Selection Prompt Design
|
|
|
+
|
|
|
+**Date:** 2026-07-15
|
|
|
+**Status:** approved for implementation planning
|
|
|
+
|
|
|
+## Context
|
|
|
+
|
|
|
+In `dual_agent` mode, ChatAgent selects EventAgent capabilities by appending
|
|
|
+event names inside an `<agent_events>` block after its visible reply. The
|
|
|
+current generated prompt explains the catalog and output format, but treats
|
|
|
+event selection as optional and does not define when an event is required.
|
|
|
+
|
|
|
+Recent local audit evidence shows the practical failure mode: ChatAgent says it
|
|
|
+will adjust volume or check weather, but emits no event. Examples include
|
|
|
+`声音调到90`, `调大一点音量`, and later weather requests. The model therefore
|
|
|
+produces a plausible conversational promise without causing the real action.
|
|
|
+
|
|
|
+The generated prompt currently exists in two equivalent construction paths:
|
|
|
+
|
|
|
+- `ToolRegistry.chat_event_system_message()` supplies the runtime fallback for
|
|
|
+ API clients that did not submit an EventPrompt.
|
|
|
+- `buildAvailableEventsPrompt()` generates the read-only EventPrompt shown and
|
|
|
+ submitted by the browser console.
|
|
|
+
|
|
|
+Both paths must retain the same behavioral contract. The text-event parser and
|
|
|
+the `<agent_events>` wire format are already stable and are not the source of
|
|
|
+the observed missed triggers.
|
|
|
+
|
|
|
+## Goals
|
|
|
+
|
|
|
+- Balance event recall against false positives.
|
|
|
+- Require an event whenever ChatAgent promises or claims a supported external
|
|
|
+ action.
|
|
|
+- Trigger low-risk, read-only, or reversible capabilities from clear natural
|
|
|
+ intent, even when the user does not phrase it as an imperative.
|
|
|
+- Ask one concise clarification before ambiguous or incomplete higher-impact
|
|
|
+ actions.
|
|
|
+- Keep ordinary conversation, hypothetical discussion, unsupported requests,
|
|
|
+ and capability questions event-free.
|
|
|
+- Preserve the existing event block protocol, parser, EventAgent argument
|
|
|
+ resolution, event kernel, and invocation-mode boundaries.
|
|
|
+- Continue recognizing old generated EventPrompts when restoring sessions or
|
|
|
+ workspace snapshots.
|
|
|
+
|
|
|
+## Non-goals
|
|
|
+
|
|
|
+- No separate intent-classification model call.
|
|
|
+- No provider-native tool schema changes.
|
|
|
+- No EventAgent argument protocol changes; ChatAgent still emits names only.
|
|
|
+- No event-kernel, handler, session, WebSocket, or active-turn behavior change.
|
|
|
+- No guarantee that every stochastic model follows the prompt perfectly; live
|
|
|
+ evaluation measures adherence but does not replace deterministic contracts.
|
|
|
+
|
|
|
+## Approaches Considered
|
|
|
+
|
|
|
+### 1. Decision-oriented prompt only
|
|
|
+
|
|
|
+Replace the weak optional wording with explicit selection rules, consistency
|
|
|
+invariants, and compact positive and negative examples. Keep the current
|
|
|
+protocol and model-call count.
|
|
|
+
|
|
|
+This is the selected approach because it directly addresses the observed
|
|
|
+failure, has the smallest runtime scope, and can be evaluated independently
|
|
|
+before introducing new architecture.
|
|
|
+
|
|
|
+### 2. Metadata-driven trigger policies
|
|
|
+
|
|
|
+Expose trigger and confirmation metadata through the event catalog and render
|
|
|
+event-specific policy dynamically. This scales better for a large heterogeneous
|
|
|
+registry, but requires domain, catalog API, frontend, and compatibility changes.
|
|
|
+It remains a possible follow-up if prompt-only evaluation shows that generic
|
|
|
+rules are insufficient.
|
|
|
+
|
|
|
+### 3. Separate event-intent classifier
|
|
|
+
|
|
|
+Run a dedicated classification step before ChatAgent response generation. This
|
|
|
+could improve isolation but adds latency, token cost, another failure mode, and
|
|
|
+coordination complexity. It is not justified before measuring the prompt-only
|
|
|
+approach.
|
|
|
+
|
|
|
+## Selected Prompt Contract
|
|
|
+
|
|
|
+The generated EventPrompt will identify itself as protocol version 2 and use
|
|
|
+imperative rules. Its dynamic catalog remains name plus description only.
|
|
|
+
|
|
|
+The prompt must communicate these rules:
|
|
|
+
|
|
|
+1. Decide whether the user's latest intent requires any available event.
|
|
|
+2. When a supported real-world action or fresh external-data request is clear,
|
|
|
+ emit the matching event.
|
|
|
+3. If the visible reply says or implies that ChatAgent will perform, has
|
|
|
+ performed, is checking, is changing, is scheduling, or is ending something,
|
|
|
+ the same reply must append the corresponding event. ChatAgent must never
|
|
|
+ promise a supported action without emitting its event.
|
|
|
+4. Low-risk, read-only, and reversible actions may be inferred from clear
|
|
|
+ natural intent. `声音太小了` can select `device.volume.adjust`, and
|
|
|
+ `明天天气怎么样` can select `knowledge.web.search`.
|
|
|
+5. For ambiguous or incomplete higher-impact actions, ask one concise
|
|
|
+ clarification and emit no event yet. A complete and unambiguous request does
|
|
|
+ not require redundant confirmation.
|
|
|
+6. Do not emit events for ordinary conversation, hypothetical statements,
|
|
|
+ discussion about a capability, unsupported behavior, or a request that does
|
|
|
+ not match an available event.
|
|
|
+7. Multiple independent clear intents may emit multiple exact event names, one
|
|
|
+ per line.
|
|
|
+8. The event block is machine-control metadata, not part of the visible reply.
|
|
|
+ It is allowed even when another prompt prohibits Markdown, lists, JSON, or
|
|
|
+ other visible formatting.
|
|
|
+9. When no event is selected, omit the event block entirely.
|
|
|
+10. Never include parameters, explanations, bullets, code fences, or invented
|
|
|
+ names inside the block.
|
|
|
+
|
|
|
+The generated shape will remain:
|
|
|
+
|
|
|
+```text
|
|
|
+Event selection protocol v2.
|
|
|
+Available events:
|
|
|
+- event.name: description
|
|
|
+...
|
|
|
+[decision rules and compact examples]
|
|
|
+First write the user-facing reply normally. If one or more events are selected,
|
|
|
+append exactly this block after the visible reply:
|
|
|
+<agent_events>
|
|
|
+event_name
|
|
|
+</agent_events>
|
|
|
+Use exact available event names only, one per line. Do not include parameters.
|
|
|
+```
|
|
|
+
|
|
|
+## Decision Examples
|
|
|
+
|
|
|
+The prompt will include a small, generic set of examples that match the current
|
|
|
+production-shaped capabilities without teaching argument extraction:
|
|
|
+
|
|
|
+| User intent | Expected selection behavior |
|
|
|
+| --- | --- |
|
|
|
+| `声音太小了` | Reply naturally and emit `device.volume.adjust`. |
|
|
|
+| `声音调到90` | Reply naturally and emit `device.volume.adjust`. |
|
|
|
+| `明天天气怎么样` | Reply naturally and emit `knowledge.web.search`. |
|
|
|
+| `你回复太慢了` | No event; response latency is not device volume. |
|
|
|
+| `你能调音量吗` | No event; this asks about capability rather than requesting action. |
|
|
|
+| `提醒我开会` | Ask for missing scheduling details and emit no event. |
|
|
|
+| `我明天下午三点要开会` | No event; this is a statement unless the user requests scheduling. |
|
|
|
+| A complete request to create a schedule | Emit `calendar.schedule.create`. |
|
|
|
+| An explicit goodbye or session-ending request | Emit `session.terminate`. |
|
|
|
+| Ordinary greeting or emotional conversation | No event. |
|
|
|
+
|
|
|
+Examples are selection guidance only. EventAgent remains responsible for
|
|
|
+arguments and validation.
|
|
|
+
|
|
|
+## Data Flow and Compatibility
|
|
|
+
|
|
|
+The browser continues to generate a read-only EventPrompt from selected tools.
|
|
|
+In `dual_agent`, exactly one managed EventPrompt is serialized. In
|
|
|
+`chat_agent_tools`, generated EventPrompt content remains excluded and provider
|
|
|
+tool schemas remain the only capability-selection surface.
|
|
|
+
|
|
|
+The runtime continues to inject its registry-generated prompt only when no
|
|
|
+event instructions are already present. The Python and JavaScript prompt
|
|
|
+builders will use the same version marker, rule ordering, examples, catalog
|
|
|
+format, and output suffix.
|
|
|
+
|
|
|
+`isGeneratedEventPromptContent()` must recognize both:
|
|
|
+
|
|
|
+- the legacy prefix `You may request EventAgent work using this text protocol.`;
|
|
|
+- the new `Event selection protocol v2.` prefix.
|
|
|
+
|
|
|
+Recognition remains conservative: a known prefix alone is insufficient. The
|
|
|
+content must also contain the event block markers and the exact-name/no-
|
|
|
+parameters instruction. This preserves cleanup of historical generated prompts
|
|
|
+without deleting ordinary custom prompts that merely mention event syntax.
|
|
|
+
|
|
|
+## Verification Design
|
|
|
+
|
|
|
+Implementation will use TDD.
|
|
|
+
|
|
|
+Deterministic tests will first fail against the current prompt and then prove:
|
|
|
+
|
|
|
+- the backend prompt contains the version marker, mandatory action/event
|
|
|
+ consistency rule, low-risk inference rule, higher-impact clarification rule,
|
|
|
+ negative boundaries, exact output block, and compact examples;
|
|
|
+- the frontend prompt contains the same behavioral anchors and catalog output;
|
|
|
+- old and new generated EventPrompt signatures are both recognized;
|
|
|
+- ordinary custom prompts containing partial markers remain retained;
|
|
|
+- payload singleton and invocation-mode filtering behavior remain unchanged;
|
|
|
+- runtime fallback injection still supplies exactly one catalog and no provider
|
|
|
+ tools in `dual_agent`;
|
|
|
+- parser behavior is unchanged.
|
|
|
+
|
|
|
+Focused verification commands will cover the prompt/runtime tests and frontend
|
|
|
+source/behavior tests, followed by JavaScript syntax, the full suite, app import,
|
|
|
+and diff checks.
|
|
|
+
|
|
|
+If a local live provider target and API key are available, run a fresh A/B
|
|
|
+evaluation with the same model and parameters. At minimum, evaluate the ten
|
|
|
+decision examples above across old and new prompts and record both missed
|
|
|
+required events and false positives. Existing historical SQLite sessions are
|
|
|
+evidence for the problem but must not be reused as an A/B input source because
|
|
|
+their accumulated context and prompt order differ.
|
|
|
+
|
|
|
+## Risks and Controls
|
|
|
+
|
|
|
+- Stronger rules may increase false positives. Negative examples and the
|
|
|
+ capability-question/hypothetical boundaries control this risk.
|
|
|
+- Generic risk language may not scale to future events. If live evaluation
|
|
|
+ exposes event-specific ambiguity, follow up with metadata-driven policy
|
|
|
+ rather than continually expanding a monolithic prompt.
|
|
|
+- Prompt duplication between Python and JavaScript can drift. Tests assert the
|
|
|
+ shared behavioral anchors and exact protocol suffix in both paths.
|
|
|
+- Historical snapshots may contain the old generated prompt. Dual-signature
|
|
|
+ detection preserves deduplication and outbound singleton guarantees.
|
|
|
+- Other system prompts may prohibit visible formatting. The v2 prompt explicitly
|
|
|
+ defines the event block as hidden machine metadata so those instructions do
|
|
|
+ not suppress event emission.
|
|
|
+
|
|
|
+## Success Criteria
|
|
|
+
|
|
|
+- Direct or naturally expressed volume requests and fresh-data questions cause
|
|
|
+ the matching event instead of an unsupported verbal promise.
|
|
|
+- Response-latency complaints, capability questions, ordinary chat, and
|
|
|
+ incomplete scheduling requests do not produce unrelated events.
|
|
|
+- No protocol, parser, provider-tool mode, EventAgent argument, or event-kernel
|
|
|
+ regression occurs.
|
|
|
+- Old and new EventPrompt snapshots remain safely deduplicated.
|
|
|
+- Deterministic verification passes, and any available live A/B evidence records
|
|
|
+ both recall and false-positive outcomes rather than reporting only successful
|
|
|
+ triggers.
|
zhenyu.hu