瀏覽代碼

docs: plan event fallback semantic fixes

Problem: fallback triggered on complete-invalid arguments, raw JSON comparison was not type-sensitive, and validator failures were misclassified as user errors.

Risk: stricter fallback semantics may reduce LLM retries that previously masked deterministic resolver or plugin-definition bugs.
zhenyu.hu 2 周之前
父節點
當前提交
7da94e3fcf
共有 2 個文件被更改,包括 60 次插入0 次删除
  1. 59 0
      docs/plans/todo-45.2-event-fallback-semantics.md
  2. 1 0
      docs/plans/todos.md

+ 59 - 0
docs/plans/todo-45.2-event-fallback-semantics.md

@@ -0,0 +1,59 @@
+# Todo 45.2 Event Fallback Semantics Plan
+
+**Status:** in_progress
+
+## Goal
+
+Make fallback and validation semantics suitable for latency/cost experiments: invoke LLM fallback only for incomplete deterministic resolution, compare raw and parsed JSON with type sensitivity, and distinguish plugin-definition failures from user argument errors.
+
+## Scope
+
+- Fall back only when required information is missing or a structured resolver explicitly marks its result incomplete.
+- Complete but invalid arguments return `INVALID_ARGUMENTS` without an LLM call.
+- Compare fallback raw JSON and parsed arguments canonically with JSON type sensitivity and reject non-standard JSON values.
+- Add `DEFINITION_ERROR` for validator/schema infrastructure failures.
+- Deep-copy/freeze registered schemas so post-registration mutation cannot bypass validation.
+- Add focused regression tests for each quality-review finding.
+
+## Files
+
+- Modify: `src/agent_lab/application/events/models.py`
+- Modify: `src/agent_lab/application/events/registry.py`
+- Modify: `src/agent_lab/application/events/kernel.py`
+- Modify: `src/agent_lab/application/tools.py` only for status compatibility mapping
+- Modify: `tests/test_event_kernel.py`
+- Modify: `tests/test_event_agent.py` only if fallback contracts change
+
+## Plan
+
+### Plan
+
+1. Add RED tests for complete-invalid no-fallback, JSON type-sensitive mismatch, broken `$ref`, and schema mutation.
+2. Verify each fails for the reviewed reason.
+3. Implement explicit resolution completeness, canonical strict JSON consistency, definition-error status, and registry schema copying.
+4. Run focused and full regressions.
+
+### Act
+
+- Test resolver output with all required fields present but wrong type/enum/additional property; fallback call count must remain zero.
+- Test missing required fields; fallback remains exactly one call.
+- Test raw `true` vs parsed `1`, raw `1` vs parsed `1.0`, nested mismatches, NaN, and Infinity; reject inconsistent/non-standard data.
+- Test an unresolved or failing `$ref`; return `DEFINITION_ERROR`, not `INVALID_ARGUMENTS`.
+- Register a definition, mutate the caller-owned schema afterward, and prove registry validation behavior is unchanged.
+- If structured deterministic resolution is introduced, preserve plain-dict resolver compatibility and require an explicit incomplete flag only for optional-schema cases.
+- Run RED, minimum implementation, focused/full verification.
+
+### Observe
+
+Confirm fallback counts now reflect missing information rather than arbitrary validation failures, raw/parsed values remain auditable, and plugin configuration failures are distinguishable from user mistakes.
+
+### Update
+
+- Mark 45, 45.1, and 45.2 `done` only after renewed reviews.
+- Commit separately with latency/cost and definition-error risks in the body.
+
+## Verification
+
+- `uv run pytest tests/test_event_kernel.py tests/test_event_agent.py -q`
+- `uv run pytest`
+- `git diff --check`

+ 1 - 0
docs/plans/todos.md

@@ -76,6 +76,7 @@
 | 44.1 | done | `docs/plans/todo-44.1-tool-transcript-hardening.md` | Harden transcript identity, reserved payload fields, EventAgent history projection, and public pre-message boundaries. | Duplicate IDs, invalid choices, reserved overrides, projected history, and pre-message boundaries are covered; full suite passes. |
 | 45 | in_progress | `docs/plans/todo-45-generic-event-kernel.md` | Build a generic event kernel with definition metadata, deterministic-first resolution, policies, execution ports, and result aggregation. | Kernel contract tests prove orchestration contains no built-in event-name branches and plugins share one pipeline. |
 | 45.1 | in_progress | `docs/plans/todo-45.1-event-kernel-boundary-hardening.md` | Harden JSON Schema validation, normalized failures, fallback raw arguments, async handlers, and existing EventAgent concurrency. | Standard schema tests and failure-path regressions pass; async handlers work through the kernel without sync-facade warnings. |
+| 45.2 | in_progress | `docs/plans/todo-45.2-event-fallback-semantics.md` | Restrict fallback to incomplete resolution, enforce JSON-type-sensitive raw consistency, and distinguish definition failures. | Complete-invalid arguments skip LLM; raw/parsed mismatch and broken schema references produce stable non-user-error results. |
 | 46 | pending | `docs/plans/todo-46-configurable-tool-invocation-mode.md` | Add `dual_agent` and `chat_agent_tools` runtime modes on the shared event kernel. | Runtime tests prove mode-specific requests, direct execution, valid provider history, and dual-mode regression safety. |
 | 47 | pending | `docs/plans/todo-47-built-in-event-plugins.md` | Add terminate-session, device-volume, schedule, and web-search as built-in plugins on the generic event kernel. | Plugin contract tests cover valid, ambiguous, invalid, idempotent, and failure results for all four events. |
 | 48 | pending | `docs/plans/todo-48-parallel-event-batches.md` | Execute multiple flat events with bounded parallelism, failure isolation, aggregation, and a terminal barrier. | Runtime tests prove independent concurrency, stable result order, isolated failures, and termination after sibling completion or timeout. |