Skip to content

fix(mcp): reword explore guidance as advisory, not a quota - #1570

Open
rongbc wants to merge 1 commit into
colbymchenry:mainfrom
rongbc:main
Open

fix(mcp): reword explore guidance as advisory, not a quota#1570
rongbc wants to merge 1 commit into
colbymchenry:mainfrom
rongbc:main

Conversation

@rongbc

@rongbc rongbc commented Aug 19, 2026

Copy link
Copy Markdown

PR Description — Reword the codegraph_explore budget note so agents read it as advisory, not a quota


Summary

This PR fixes a wording problem in the codegraph_explore response footer. The server appends a note like
Explore budget: 2 calls for this project (3,497 files indexed) … Synthesize once you've used 2 to every
response on mid-sized projects. My AI coding assistant read that as a hard quota, stopped calling codegraph_explore
after two invocations, and fell back to grep + Read — which cost more tokens and produced worse context than a
third explore would have. The note was meant as soft guidance; nothing in it said so. This PR rewords it to be
explicitly advisory ("NOT a quota", "extra calls are never rejected"), and updates the eval probe and tests that
pin the old wording.

Background

codegraph_explore returns a bounded slice of source per call, tuned by project size
(getExploreOutputBudget / getExploreBudget in src/mcp/tools.ts, introduced in #185). On small-to-medium
repositories the intent is to keep a single response tight — under the host's inline tool-result ceiling so the
agent doesn't have to Read an externalized file back — and to steer the agent toward a few focused calls instead
of one sprawling query.

The steering takes the form of a footer note appended to every response when the project sits in the
500–4,999-file tier (3,497 files indexed for my project, so this tier is where I hit it). The original text:

Explore budget: 2 calls for this project (3,497 files indexed). Each call covers ~6 files; if your
question spans more, spend your remaining calls on the uncovered area BEFORE falling back to Read — another
explore is cheaper and more complete than reading those files. Synthesize once you've used 2.

Why it scared my AI assistant

The note is generated by the server at src/mcp/tools.ts:5696 and is purely informational — the server never
enforces or rate-limits explore calls, and a third call always succeeds. But the wording reads like an enforced
budget, and that is exactly how my assistant treated it:

  1. The word "budget". In a tooling context, "budget" implies a quota or ceiling. Combined with
    spend your remaining calls, the calls sound like a depleting allowance.
  2. The imperative close. Synthesize once you've used 2 is a direct order to wrap up after two calls.
    LLM agents tend to comply with imperative text in tool output as if it were a runtime constraint.
  3. No negation anywhere. Nothing in the note says "advisory", "not enforced", or "extra calls will still
    work". An agent cannot tell soft guidance from a hard cap unless the text says so explicitly.
  4. The number contrast. 2 next to 3,497 files indexed reinforces the intuition that explore calls are
    expensive and rationed.
  5. It conflicted with my repo's working rules. My AGENTS.md says code-understanding questions must start
    with codegraph_explore and use grep only as a supplement. When the budget note (stop after 2) and the
    rule (keep exploring) pointed in opposite directions, the assistant obeyed the one that looked like a system
    constraint — the tool's own quota-flavored text.

What actually happened (2026-08-19, reproducible): my assistant used two explores on my project, saw
Explore budget: 2 calls, assumed the allowance was spent, and switched the follow-up query to grep + Read —
reading whole files (agent_api.c, wifi_service.c, …) that one more explore would have returned more cheaply
and more completely. A third explore still succeeded, proving there was never any enforcement. The only reason
the assistant behaved that way is that the note did not say "this is a suggestion".

Fix overview

  • Reword the footer (src/mcp/tools.ts:5696): Exploration guidance — advisory only, NOT a quota;
    explicitly state "extra calls are never rejected or rate-limited"; replace the imperative
    Synthesize once you've used N with a conditional "only stop exploring when the response actually covers
    the flow you asked about".
  • Optional hardening: only emit the note when the response was actually trimmed (so the guidance appears
    exactly when more calls are genuinely useful), and consider dropping the call count entirely.
  • Update the pinned consumers of the old wording: scripts/agent-eval/probe-suite-envelope.mjs:88 and
    __tests__/explore-output-budget.test.ts (including a new regression assertion that the tier-2 note
    contains "NOT a quota" and never reintroduces "remaining calls" / "Synthesize once").
  • Regenerate dist/mcp/tools.js via npm run build.

Tests

  • npm run build && npm test (vitest: explore-output-budget.test.ts, explore-allocation-e2e.test.ts).
  • Manual smoke: codegraph.js serve --mcp --path <project> on a 3,497-file project, confirm the footer
    carries the new advisory wording.

The trailing explore note said 'Explore budget: N calls ... spend your remaining calls ... Synthesize once you've used N' — agents read it as a hard quota, stopped exploring early, and fell back to grep+Read (more tokens, less complete). Reword it as explicit advisory guidance: the server never rejects or rate-limits extra explore calls, and exploring further is encouraged when the response doesn't cover the flow.

The tool schema description carried the same quota language ('Budget: make at most N calls for this project') — the higher-salience text the agent reads before ever calling. Reword it identically: advisory only, NOT a quota.

Also sync the eval probe string (probe-suite-envelope.mjs) and pin the new wording in tests with anti-regression assertions (medium tier must contain 'NOT a quota' and never 'remaining calls'/'Synthesize once'/'make at most').
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant