[czenko]

Part of DOCS-1473.

Problem

The docs-nav suite needs to measure whether an agent can find the docs page that answers a task by following links on supabase.com. The agent must have no search and no way to open a URL it remembers.

This PR sets the scaffolding for this kind of eval, so it asks for an AI team review.

Solution

apps/framework/
├── package.json                    # pnpm test runs test:docs-nav-lib
└── vitest.docs-nav-lib.config.ts   # runs the docs-nav lib tests from the repo root
experiments/docs-nav/lib/
├── docs-navigator.mjs              # MCP server with one tool, open_page
└── docs-nav-experiment.ts          # Claude Code, Sonnet 5.5, medium effort, tools: [], navigator only
evals/docs-nav/lib/
└── navigation.ts                   # scoreDocsNav: reached, wrong turns, judged, grounded

• Navigator: opens the docs root and the links on pages the agent has opened, and refuses anything else.
  • An experiment can set the docs IA variant. The navigator adds ?docs-ia-mvp-v1=<variant> to every request, and the opened line shows the URL as fetched.
  • In nav-only mode, it lists only links inside data-docs-nav regions, so page bodies can't shortcut the tree.
  • It always reads production docs on supabase.com.
• Scorer: loads /docs/nav-tree.json for the variant in the opened URL.
  • A run passes when it reaches a target in no more hops than the tree has levels.
  • A page opened before the target that isn't a target or one of its tree ancestors is a wrong turn. No wrong turns is the headline measure.
  • Control runs also accept control targets, for tasks whose answer is a section of another page in today's nav.
  • When no opened page matches a target, a judge reads each opened page in order, within the hop limit, and the run is marked judged.
  • A grounded check, reported apart from the pass, judges whether the answer agrees with the reached page.
• Agent: the same model and effort as the repo's Claude Code preset.
• CI: pnpm test runs the docs-nav lib tests, the same way test:cli-lib runs the CLI suite's.

The tree route and the data-docs-nav markers come from supabase/supabase#52735. Runs can't be scored until it ships.

Manual testing

• pnpm typecheck and pnpm test pass.
• pnpm --filter @supabase-evals/framework test:docs-nav-lib runs the 26 lib tests, and they pass.
• Broke each new scoring rule in turn (tree ancestors, the variant's tree, the depth limit, control targets) and nav-only links. A test failed each time.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Nov 3, 2026 11:27pm UTC