소스 검색

fix: isolate event tool failures

zhenyu.hu 3 주 전
부모
커밋
2ff8aec5d8
5개의 변경된 파일103개의 추가작업 그리고 7개의 파일을 삭제
  1. 10 5
      docs/plans/todo-16-tool-error-isolation.md
  2. 1 1
      docs/plans/todos.md
  3. 8 1
      src/agent_lab/application/event_agent.py
  4. 49 0
      tests/test_debug_runtime.py
  5. 35 0
      tests/test_event_agent.py

+ 10 - 5
docs/plans/todo-16-tool-error-isolation.md

@@ -1,6 +1,6 @@
 # Todo 16 Tool Error Isolation Plan
 
-**Status:** in_progress
+**Status:** done
 
 ## Goal
 
@@ -53,8 +53,13 @@ uv run pytest
 
 ## Verification
 
-Expected result:
+Result:
 
-- Direct EventAgent tests prove structured tool errors.
-- Runtime tests prove the WebSocket stream contract stays structured.
-- Full suite passes.
+- Red run passed as expected: direct EventAgent raised `RuntimeError("boom")`, and runtime output collection timed out waiting for a missing tool reply.
+- `uv run pytest tests/test_event_agent.py tests/test_debug_runtime.py` passed: 14 tests.
+- `uv run pytest` passed: 37 tests, 1 existing Starlette deprecation warning.
+
+## Evaluation
+
+- `EventAgent.handle()` now catches tool handler `Exception` and returns a structured tool reply with `{"tool": name, "error": "tool handler failed: ..."}`.
+- Runtime receives that reply through the input queue, emits `tool_result`, and continues to the next ChatAgent call.

+ 1 - 1
docs/plans/todos.md

@@ -42,4 +42,4 @@
 | 13 | done | `docs/plans/todo-13-real-queue-websocket-loop.md` | Make runtime queues real producer/consumer boundaries and let WebSocket run separate upstream/downstream tasks. | `uv run pytest tests/test_debug_runtime.py tests/test_websocket_api.py`; `uv run pytest`. |
 | 14 | done | `docs/plans/todo-14-message-role-validation.md` | Reject provider-invalid API message roles and malformed tool replies before network calls. | `uv run pytest tests/test_websocket_api.py tests/test_event_agent.py`; `uv run pytest`. |
 | 15 | done | `docs/plans/todo-15-static-package-data.md` | Include static UI assets in installed package builds. | `uv run pytest tests/test_packaging.py`; `uv run pytest`; `uv build`; wheel inspection. |
-| 16 | in_progress | `docs/plans/todo-16-tool-error-isolation.md` | Convert EventAgent tool handler exceptions into structured tool-result errors instead of aborting the session. | Tests prove a failing tool returns an error payload and the WebSocket run stays structured. |
+| 16 | done | `docs/plans/todo-16-tool-error-isolation.md` | Convert EventAgent tool handler exceptions into structured tool-result errors instead of aborting the session. | `uv run pytest tests/test_event_agent.py tests/test_debug_runtime.py`; `uv run pytest`. |

+ 8 - 1
src/agent_lab/application/event_agent.py

@@ -26,7 +26,14 @@ class EventAgent:
                 },
             )
 
-        return self._tool_reply(event, self.registry.handle(event))
+        try:
+            payload = self.registry.handle(event)
+        except Exception as exc:
+            payload = {
+                "tool": event.name,
+                "error": f"tool handler failed: {exc}",
+            }
+        return self._tool_reply(event, payload)
 
     def _tool_reply(self, event: ToolCallEvent, payload: dict[str, Any]) -> ChatMessage:
         return ChatMessage(

+ 49 - 0
tests/test_debug_runtime.py

@@ -1,5 +1,6 @@
 import asyncio
 import importlib
+import json
 from collections.abc import AsyncIterator
 from typing import Any
 
@@ -17,6 +18,10 @@ def _runtime_queues_class():
     return module.RuntimeQueues
 
 
+async def _collect_outputs(stream: AsyncIterator[dict[str, Any]]) -> list[dict[str, Any]]:
+    return [message async for message in stream]
+
+
 class RecordingQueue(asyncio.Queue):
     def __init__(self, name: str, log: list[tuple[str, str, str]]) -> None:
         super().__init__()
@@ -261,6 +266,50 @@ async def test_runtime_buffers_upstream_user_input_until_after_matching_tool_rep
     assert client.second_call_messages[3].content == "follow-up while tool runs"
 
 
+@pytest.mark.asyncio
+async def test_runtime_continues_when_event_agent_tool_handler_raises():
+    def fail_tool(event: ToolCallEvent) -> dict[str, Any]:
+        raise RuntimeError("boom")
+
+    registry = ToolRegistry(
+        [
+            ToolDefinition(
+                name="handoff_note",
+                description="Broken handoff tool.",
+                parameters={"type": "object"},
+                handler=fail_tool,
+            )
+        ]
+    )
+    request = DebugRunRequest(
+        user_message="debug this",
+        system_prompts=[],
+        pre_messages=[],
+        chat_agent=AgentParams(model="fake-model", temperature=0.1, max_tokens=200),
+        event_agent=EventAgentParams(enabled_tools=["handoff_note"], max_event_loops=2),
+    )
+    runtime = DebugRuntime(FakeChatClient(), registry=registry)
+
+    outputs = await asyncio.wait_for(
+        _collect_outputs(runtime.run(request)),
+        timeout=1,
+    )
+
+    assert [message["type"] for message in outputs] == [
+        "session_started",
+        "event",
+        "tool_result",
+        "round_stats",
+        "message_delta",
+        "round_stats",
+        "done",
+    ]
+    assert json.loads(outputs[2]["message"]["content"]) == {
+        "tool": "handoff_note",
+        "error": "tool handler failed: boom",
+    }
+
+
 @pytest.mark.asyncio
 async def test_runtime_uses_event_and_input_queues_for_event_agent_handoff():
     RuntimeQueues = _runtime_queues_class()

+ 35 - 0
tests/test_event_agent.py

@@ -72,3 +72,38 @@ async def test_event_agent_returns_registry_errors_for_disabled_and_unknown_tool
         "tool": "missing_tool",
         "error": "unknown tool",
     }
+
+
+@pytest.mark.asyncio
+async def test_event_agent_returns_structured_error_when_tool_handler_raises():
+    def fail_tool(event: ToolCallEvent) -> dict:
+        raise RuntimeError("boom")
+
+    registry = ToolRegistry(
+        [
+            ToolDefinition(
+                name="handoff_note",
+                description="Send a note to the event agent.",
+                parameters={"type": "object"},
+                handler=fail_tool,
+            )
+        ]
+    )
+    event = ToolCallEvent(
+        id="call_1",
+        name="handoff_note",
+        arguments={"message": "inspect this event"},
+        raw_arguments='{"message":"inspect this event"}',
+    )
+
+    reply = await EventAgent(
+        enabled_tools=["handoff_note"],
+        registry=registry,
+    ).handle(event)
+
+    assert reply.role == "tool"
+    assert reply.tool_call_id == "call_1"
+    assert json.loads(reply.content) == {
+        "tool": "handoff_note",
+        "error": "tool handler failed: boom",
+    }