Skip to content

fix(trust-gate): prevent propose_truth from hanging indefinitely on LLM timeout - #19

Merged
timeleft-- merged 4 commits into
mainfrom
fix/propose-truth-hang
Mar 10, 2026
Merged

timeleft-- merged 4 commits into
mainfrom
fix/propose-truth-hang

Conversation

@timeleft--

Copy link
Copy Markdown
Member

Summary

  • Root cause: propose_truth calling OpenRouter via asyncio.wait_for with 60s timeout, but client_args={"timeout": scalar} left the httpx pool timeout unset (infinite). OpenRouter accepted the connection then never responded — 12+ hour hang in session b3147494.
  • Fix 1 (critical): @with_tool_timeout decorator on handle_call_tool — wraps every MCP tool call in asyncio.wait_for, reads tool_timeout_secs from config at call time. Default: 300s.
  • Fix 2 (critical): asyncio.wait_for around the Trust Gate LLM call specifically, using configurable trust_gate_timeout_secs. Default: 120s. Returns structured error on timeout instead of hanging indefinitely.
  • Fix 3 (defense in depth): Explicit httpx.Timeout(connect=10, read=N, write=N, pool=10) — sets all httpx timeout phases explicitly so no phase defaults to infinite.
  • Fix 4 (observability): Rotating file handler on MCP server logger → ~/.fava-trails/logs/mcp-server.log. Without this, server-side hangs are undiagnosable since Claude Code doesn't capture MCP stderr to disk.

Configuration

Both timeouts are configurable in config.yaml, set to 0 to disable:

tool_timeout_secs: 300      # outermost guard on every tool call
trust_gate_timeout_secs: 120  # Trust Gate LLM call guard

Test plan

  • test_propose_truth_timeout_returns_error — Trust Gate LLM timeout returns structured error
  • TestWithToolTimeout::test_timeout_returns_error_dict — decorator fires after configurable timeout
  • TestWithToolTimeout::test_passes_through_on_success — no interference on normal calls
  • TestWithToolTimeout::test_disabled_when_zero — timeout disabled when tool_timeout_secs=0
  • 420/420 tests pass

🤖 Generated with Claude Code

Comment thread tests/test_trust_gate.py Fixed
timeleft-- and others added 4 commits March 10, 2026 08:08
…LM timeout

Three-layer defense against the 12-hour hang (session b3147494):

1. asyncio.wait_for(timeout=120) wraps the Trust Gate LLM call in
   handle_propose_truth — fires after 2 minutes and returns a structured
   error, preventing infinite MCP server block regardless of provider behavior.

2. Explicit httpx.Timeout decomposition in LLMClient.chat — replaces the
   scalar timeout with httpx.Timeout(connect=10, read=N, write=N, pool=10)
   so all httpx timeout phases are explicitly bounded (scalar timeout leaves
   the pool timeout as None/infinite).

3. Rotating file handler for MCP server logs — persists to
   ~/.fava-trails/logs/mcp-server.log so server-side hangs are diagnosable
   without relying on Claude Code session.jsonl.

Adds regression test: test_propose_truth_timeout_returns_error.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add `tool_timeout_secs` (default 300) and `trust_gate_timeout_secs`
  (default 120) to GlobalConfig — set either to 0 to disable
- Introduce `@with_tool_timeout` decorator on `handle_call_tool`:
  wraps every tool dispatch in asyncio.wait_for, reads config at
  call time so changes don't require a server restart
- Replace hardcoded 120.0 in handle_propose_truth with
  `global_config.trust_gate_timeout_secs`
- Add tests: decorator pass-through, timeout error dict, disabled-when-zero

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- HIGH: honour trust_gate_timeout_secs=0 as "disabled" in handle_propose_truth
  (previously always called asyncio.wait_for, raising immediately on 0.0)
- MEDIUM: wrap rotating file handler setup in try/except OSError so a
  restricted filesystem never prevents server startup (falls back to stderr)
- MEDIUM: lock _get_trail with asyncio.Lock + double-check to prevent
  double-init race when parallel calls create the same new scope simultaneously
- LOW: add httpx>=0.27.0 as explicit dependency in pyproject.toml
- LOW: add model_validator to GlobalConfig enforcing trust_gate_timeout_secs <
  tool_timeout_secs (when both > 0) with NonNegativeInt constraints on both
- Tests: validator happy/sad paths, disable-when-zero path, lock race coverage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@timeleft--
timeleft-- force-pushed the fix/propose-truth-hang branch from 55d439b to 4ec16b4 Compare March 10, 2026 12:09
@timeleft--
timeleft-- merged commit a2f8840 into main Mar 10, 2026
7 checks passed
@timeleft--
timeleft-- deleted the fix/propose-truth-hang branch March 10, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant