Skip to content

fix(appkit): export MLflow traces when the request span has a remote parent - #717

Open
fjakobs wants to merge 1 commit into
mainfrom
fjakobs/la-2052-mlflow-traces-deployed
Open

fjakobs wants to merge 1 commit into
mainfrom
fjakobs/la-2052-mlflow-traces-deployed

Conversation

@fjakobs

@fjakobs fjakobs commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Why

  • Agent traces never reach the MLflow experiment from an app deployed on Databricks Apps. The same app run locally does export them.
  • The Apps proxy sends a W3C traceparent header with every request. AppKit's tracer provider picks up that context, so the first span of each request (the HTTP server span) has a remote parent (parentSpanContext.isRemote === true).
  • @mlflow/core (0.4.0, and master as of today) treats a span as the trace root only if it has no parent at all. It ignores isRemote. See exporters/mlflow.js:36/86/151 and exporters/uc_table.js:38/89/153 in the published build.
  • As a result, mlflow never registers a trace for the request, and every span logs No trace ID found for span <name>. Skipping. Nothing is exported. GatedMlflowSpanProcessor had the same root check.
  • The No trace ID found warnings on plain (non-agent) requests have the same cause.
  • Upstream report: [BUG] TypeScript tracing drops every trace whose root span has a remote parent (incoming traceparent) mlflow/mlflow#27545.

Change

  • In GatedMlflowSpanProcessor (packages/appkit/src/plugins/agents/mlflow.ts), a span whose parent is remote now counts as the root of this process's trace.
  • The gate passes mlflow a view of that span with no parent. The view is a Proxy that hides parentSpanContext and binds methods to the real span, so mlflow's writes such as setAttribute land on the real span. The real span is not changed and keeps its remote parent, so OTLP export still joins the caller's distributed trace.
  • The gate now records which object it forwarded (WeakSet became WeakMap), so onStart and onEnd hand mlflow the same view. The outbound-CLIENT drop and the root check both use the new definition of root.
  • The fix covers both the classic experiment processor and the Unity Catalog table processor. It is marked as a shortcut:, to be removed once upstream treats remote parents as roots.

Testing

  • New unit tests in plugins/agents/tests/mlflow.test.ts. All 3 fail on main and pass with this change:
    • A deployed agent turn whose request span has a remote parent is exported. mlflow gets one parentless view at start and at end, and the real span keeps its remote parent.
    • A deployed plain request with a remote parent is discarded (popTrace). It is not left unregistered.
    • With a real SDK span started under a remote context, the view writes through to the real span and reads its live state.
  • pnpm build, pnpm docs:build, pnpm check:fix, pnpm -r typecheck and pnpm test all pass.
  • End to end against a Databricks workspace, using a minimal app (server() + agents() with one markdown agent on a foundation model endpoint, and an MLflow experiment bound through the agents plugin's experiment resource). The SDK was packed from this branch.
Run Before After
Local, no traceparent trace exported trace exported
Local, with traceparent no trace, No trace ID found for span POST trace exported, no warnings
Deployed on Databricks Apps (proxy sends traceparent) no trace, No trace ID found for POST and the agent span trace exported (mlflow_trace_id in the response), no warnings
Plain analytics request with traceparent No trace ID found for every span no trace, no warnings
  • The exported trace has the right shape. The root is POST /invocations with an empty parent_span_id, and the agent span sits under it. Locally, the MLflow trace id matched the caller's traceparent trace id.
  • On the deployed app, non-agent requests still create no traces, and the trace count stayed stable after the requests, so the exporters' own uploads don't loop back into traces.

This pull request and its description were written by Isaac.

…parent

Deployed apps never exported agent traces. The Apps proxy forwards a W3C
`traceparent`, so each request's first span has a remote parent, and
@mlflow/core (0.4.0) roots a trace only at a span with no parent at all:
no trace was registered and every span logged "No trace ID found for span
... Skipping." AppKit's GatedMlflowSpanProcessor made the same root check.

Treat a remote-parent span as the local root: the gate hands mlflow a
parentless view of it (a Proxy that hides `parentSpanContext` and binds
methods to the real span), while the real span keeps its remote parent for
OTLP. Covers both the classic and the UC processor, and also silences the
"No trace ID found" warnings for plain requests that carry `traceparent`.

Upstream: mlflow/mlflow#27545

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Fabian Jakobs <fabian.jakobs@databricks.com>
@fjakobs
fjakobs requested a review from a team as a code owner November 6, 2026 15:30
@fjakobs
fjakobs requested a review from ditadi November 6, 2026 15:30
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.2 MB (+928 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.3 MB (+998 B) 460 KB (+386 B)
Type declarations 465 KB 169 KB (-1 B)
Source maps 2.5 MB (+1.6 KB) 859 KB (+589 B)
Other 11 KB 3.7 KB
Total 4.2 MB (+2.6 KB) 1.5 MB (+974 B)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 106 KB 2.5 KB 109 KB external 344 KB
./beta 103 KB (+62 B) 482 B 104 KB (+62 B) external 310 KB (+212 B)
./testing 45 KB 32 KB 77 KB external 223 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB 0 B 26 KB external 74 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 102 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 83 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.2 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 647 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 176 B
./beta index.js lazy 114 B
./testing manifest.js initial 32 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 28 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB (-4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 429 KB 147 KB
Type declarations 267 KB 98 KB (-1 B)
Source maps 853 KB 286 KB
CSS 16 KB 3.2 KB
Total 1.5 MB 534 KB (-1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 433 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 4.9 KB 0 B 4.9 KB 0 B 12 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 431 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 4.9 KB

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 42243687946 -R databricks/appkit -n appkit-template-0.90.0-pr.a54a550-fjakobs-la-2052-mlflow-traces-deployed-717 -D appkit-pr-717 \
  && unzip -o "appkit-pr-717/appkit-template-0.90.0-pr.a54a550-fjakobs-la-2052-mlflow-traces-deployed-717.zip" -d "appkit-pr-717" \
  && databricks apps init --template "appkit-pr-717"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

@fjakobs
fjakobs requested a review from MarioCadenas November 6, 2026 16:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants