[fjakobs]

Problem

appkit agent eval --url https://<app>.databricksapps.com --profile <profile> fails every eval against a deployed app. The Databricks Apps OAuth proxy rejects the chat request with a 401, so each turn ends with agent turn did not complete successfully. The only workaround was passing the token by hand:

appkit agent eval --url "$APP_URL" --profile <profile> \
  --header "Authorization: Bearer $(databricks auth token -p <profile> | jq -r .access_token)"

The CI example in the agents docs (--url "$APP_URL" --profile ci) has the same problem.

Root cause

runAgentEval already resolves a Databricks host and OAuth token from --profile, --databricks-* or DATABRICKS_*. That token only reached the MLflow and LLM-judge configs. The headers for requests to the app were built from --header alone, so the app never got a bearer.

Fix

A new helper, withAppAuth(baseUrl, headers, token), adds Authorization: Bearer <token> to the driver's headers when:

• --url is an https://*.databricksapps.com app, so the workspace token is never sent to localhost or any other host, and
• the caller hasn't set their own Authorization header with --header (matched case-insensitively). An explicit header always wins.

The token is minted once per run, the same way MLflow and the judge already use it. The --profile help text and the agents docs now say this. They also note that Databricks Apps accept only OAuth tokens, so a PAT (--databricks-token / DATABRICKS_TOKEN) still gets a 401 from a deployed app.

Tests

packages/shared/src/cli/commands/agent/eval.test.ts covers withAppAuth:

• it adds the bearer for a deployed app URL and keeps other --headers;
• an explicit authorization header wins;
• it leaves headers untouched for http://localhost, plain-http app URLs, other hosts, lookalikes (databricksapps.com.evil.com), unparseable URLs, and a missing token.

Verification

End to end against a Databricks workspace:

• I packed the SDK with pnpm pack:sdk and installed it in a scratch project with one smoke eval.
• I deployed a minimal app that serves /api/agents/chat.
• Before the fix: appkit agent eval --url <app> --profile <profile> failed with a 401 (exit 1), and passing the bearer by hand with --header passed.
• After the fix: the same command with no --header passes (exit 0). An explicit bad --header Authorization still takes precedence and fails as expected. A run against a local server with --profile passes, and that server received no Authorization header.
• I also confirmed the Apps proxy consumes the caller's Authorization header and doesn't pass it on to app code, which only gets x-forwarded-access-token. Sending the token to *.databricksapps.com is equivalent to signing in through a browser.

pnpm build, pnpm docs:build, pnpm check:fix, pnpm -r typecheck and pnpm test pass.

Not in this PR

• Refreshing the token for eval runs longer than the OAuth token lifetime (about 1h for U2M).
• Apps served from domains outside *.databricksapps.com still need --header.
• A non-2xx response from the app is still reported as a generic failed turn, without the HTTP status.

This pull request and its description were written by Isaac.

[github-actions]

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.2 MB (+7 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	1.3 MB	459 KB
Type declarations	465 KB	169 KB (+1 B)
Source maps	2.5 MB	859 KB
Other	11 KB	3.7 KB
Total	4.2 MB	1.5 MB (+1 B)

Per-entry composition (own code — deps external (as shipped))

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
.	106 KB	2.5 KB	109 KB	external	344 KB
./beta	103 KB	482 B	104 KB	external	310 KB
./testing	45 KB	32 KB	77 KB	external	223 KB
./tsdown	520 B	0 B	520 B	external	813 B
./type-generator	26 KB	0 B	26 KB	external	74 KB

Chunks:

Entry	Chunk	Load	Size (gz)
.	index.js	initial	102 KB
.	utils.js	initial	4.6 KB
.	remote-tunnel-manager.js	lazy	2.5 KB
./beta	beta.js	initial	83 KB
./beta	modular.js	initial	6.0 KB
./beta	stream-manager.js	initial	5.9 KB
./beta	service-context.js	initial	4.2 KB
./beta	databricks.js	initial	3.3 KB
./beta	client.js	initial	647 B
./beta	index.js	initial	20 B
./beta	supervisor-api.js	lazy	192 B
./beta	databricks.js	lazy	176 B
./beta	index.js	lazy	114 B
./testing	manifest.js	initial	32 KB
./testing	index.js	initial	10 KB
./testing	wide-event-emitter.js	initial	2.9 KB
./testing	index.js	lazy	28 KB
./testing	remote-tunnel-manager.js	lazy	2.5 KB
./testing	utils.js	lazy	1.8 KB
./tsdown	index.js	initial	520 B
./type-generator	index.js	initial	26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	429 KB	147 KB
Type declarations	267 KB	98 KB
Source maps	853 KB	286 KB
CSS	16 KB	3.2 KB
Total	1.5 MB	534 KB

Per-entry composition (consumer bundle — deps bundled, peerDeps external)

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
./js	5.3 KB	49 KB	55 KB	208 KB	14 KB
./js/beta	2.1 KB	0 B	2.1 KB	0 B	5.0 KB
./react	433 KB	49 KB	481 KB	1.3 MB	177 KB
./react/beta	4.9 KB	0 B	4.9 KB	0 B	12 KB

Chunks:

Entry	Chunk	Load	Size (gz)
./js	index.js	initial	5.2 KB
./js	chunk	initial	120 B
./js	apache-arrow	lazy	49 KB
./js/beta	beta.js	initial	2.1 KB
./react	index.js	initial	431 KB
./react	tslib	initial	2.1 KB
./react	apache-arrow	lazy	49 KB
./react/beta	beta.js	initial	4.9 KB

[github-actions]

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 42241887747 -R databricks/appkit -n appkit-template-0.90.0-pr.1bb8bab-fjakobs-la-2053-agent-eval-auth-711 -D appkit-pr-711 \
  && unzip -o "appkit-pr-711/appkit-template-0.90.0-pr.1bb8bab-fjakobs-la-2053-agent-eval-auth-711.zip" -d "appkit-pr-711" \
  && databricks apps init --template "appkit-pr-711"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.