agent-lab/docs/plans/todo-45.2-event-fallback-semantics.md

Todo 45.2 Event Fallback Semantics Plan

Status: done

Result

Fallback now represents missing information rather than arbitrary invalid input. Complete-invalid and provider-resolved arguments skip fallback; strict JSON type/raw consistency and definition-error separation are covered.

Goal

Make fallback and validation semantics suitable for latency/cost experiments: invoke LLM fallback only for incomplete deterministic resolution, compare raw and parsed JSON with type sensitivity, and distinguish plugin-definition failures from user argument errors.

Scope

• Fall back only when required information is missing or a structured resolver explicitly marks its result incomplete.
• Complete but invalid arguments return INVALID_ARGUMENTS without an LLM call.
• Compare fallback raw JSON and parsed arguments canonically with JSON type sensitivity and reject non-standard JSON values.
• Add DEFINITION_ERROR for validator/schema infrastructure failures.
• Deep-copy/freeze registered schemas so post-registration mutation cannot bypass validation.
• Add focused regression tests for each quality-review finding.

Files

• Modify: src/agent_lab/application/events/models.py
• Modify: src/agent_lab/application/events/registry.py
• Modify: src/agent_lab/application/events/kernel.py
• Modify: src/agent_lab/application/tools.py only for status compatibility mapping
• Modify: tests/test_event_kernel.py
• Modify: tests/test_event_agent.py only if fallback contracts change

Plan

Plan

1. Add RED tests for complete-invalid no-fallback, JSON type-sensitive mismatch, broken $ref, and schema mutation.
2. Verify each fails for the reviewed reason.
3. Implement explicit resolution completeness, canonical strict JSON consistency, definition-error status, and registry schema copying.
4. Run focused and full regressions.

Act

• Test resolver output with all required fields present but wrong type/enum/additional property; fallback call count must remain zero.
• Test missing required fields; fallback remains exactly one call.
• Test raw true vs parsed 1, raw 1 vs parsed 1.0, nested mismatches, NaN, and Infinity; reject inconsistent/non-standard data.
• Test an unresolved or failing $ref; return DEFINITION_ERROR, not INVALID_ARGUMENTS.
• Register a definition, mutate the caller-owned schema afterward, and prove registry validation behavior is unchanged.
• If structured deterministic resolution is introduced, preserve plain-dict resolver compatibility and require an explicit incomplete flag only for optional-schema cases.
• Run RED, minimum implementation, focused/full verification.

Observe

Confirm fallback counts now reflect missing information rather than arbitrary validation failures, raw/parsed values remain auditable, and plugin configuration failures are distinguishable from user mistakes.

Update

• Mark 45, 45.1, and 45.2 done only after renewed reviews.
• Commit separately with latency/cost and definition-error risks in the body.

Verification

• uv run pytest tests/test_event_kernel.py tests/test_event_agent.py -q
• uv run pytest
• git diff --check