[HDX-5162] LLM observability dashboard, span chat view, and sessions - #2990
Open
wrn14897 wants to merge 21 commits into
Open
[HDX-5162] LLM observability dashboard, span chat view, and sessions#2990wrn14897 wants to merge 21 commits into
wrn14897 wants to merge 21 commits into
Conversation
…s, and /llm dashboard
Adds read-time, schema-agnostic LLM observability on top of existing trace
and log data. No ingestion changes: everything derives from span/log
attribute maps at query time, so it works retroactively on already-ingested
telemetry.
- Normalization lib (packages/app/src/llm/lib): detects LLM spans and
normalizes model, provider, token usage (incl. cached/reasoning), cost,
session ids, TTFT, and chat messages across four instrumentation dialects:
OTel GenAI semconv (attribute- and event-based), OpenLLMetry,
OpenInference, and the Vercel AI SDK. Includes real-world variants
observed from opencode and Claude Code telemetry (whole-string
llm.input_messages, camelCase ai.usage.*, flat token/cost keys,
bracketed model ids like claude-opus-5[1m]).
- Span side panel: an LLM tab renders the normalized conversation (roles,
markdown, tool calls) with a usage/cost summary; the Overview tab gains an
LLM section; trace waterfall labels LLM spans with model + token count.
- Cost estimation: bundled model price catalog (adapted from Langfuse's
MIT-licensed price list) with regex matching for provider/Bedrock/Vertex
id flavors; an instrumentation-provided cost attribute always wins.
- /llm preset dashboard (Overview | Sessions | Search):
- Overview: KPI tiles (calls, tokens, est. cost, avg cost/call, cache hit
rate, error rate), calls/error trends, token split
(uncached/cached/output/reasoning), cost by model, finish reasons,
models/services/users/error tables, latency heatmap, p95 by model,
TTFT, and tool analytics.
- Sessions: activity grouped by the cross-dialect session id
(gen_ai.conversation.id, session.id, ai.telemetry.metadata.sessionId)
with a drawer timeline; span attributes load lazily per expanded call
since agent SDKs stamp full conversation history on every span
(~48 MiB -> ~20 KiB list payload).
- Search: side-by-side LLM trace span and log event tables, correlating
signals for instrumentations that emit session ids without trace
context.
- Aggregations gate token/cost sums on authoritative usage reporters so SDK
wrapper spans don't double count.
Known limitations: bundled prices go stale between releases, and apps that
double-instrument (e.g. opencode emitting both OpenInference and Vercel AI
spans per call) still double count in sums.
🦋 Changeset detectedLatest commit: 8741d9a The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
the
review/tier-4
Contributor
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Contributor
Greptile SummaryThis PR adds a schema-agnostic LLM observability dashboard, normalized multi-dialect telemetry extraction, session chat views, latency analysis, and optional cost estimation. It also adds Vercel AI SDK telemetry metadata for team and user attribution.
Confidence Score: 4/5The PR is not yet safe to merge because session rows without trace identity can still resolve to the wrong span detail when identifiers collide. The trace-aware lookup fixes collisions for populated trace IDs, but the session list still converts absent trace IDs to an empty string and the detail query applies LIMIT 1 to a trace ID, span ID, and timestamp tuple that can remain non-unique. Files Needing Attention: packages/app/src/llm/dashboard/LLMSessionPanel.tsx and packages/app/src/llm/dashboard/SessionSpanDetail.tsx
|
| Filename | Overview |
|---|---|
| packages/app/src/llm/dashboard/LLMDashboardPage.tsx | Coordinates source selection, URL-backed filters, time range, and the four LLM dashboard tabs. |
| packages/app/src/llm/dashboard/LLMSessionPanel.tsx | Loads lightweight session span rows and passes their trace, span, and timestamp fields to lazy detail views. |
| packages/app/src/llm/dashboard/SessionSpanDetail.tsx | Fetches expanded span attributes using trace, span, and timestamp predicates, but the lookup remains non-unique when trace identity is absent. |
| packages/app/src/llm/lib/expressions.ts | Builds cross-dialect ClickHouse expressions for normalized LLM fields, usage election, and cost calculations. |
| packages/app/src/llm/lib/extract.ts | Extracts normalized LLM span fields and messages from supported telemetry dialects. |
| packages/api/src/controllers/ai.ts | Adds a helper that enables Vercel AI SDK telemetry and removes nullish metadata values. |
| packages/app/src/components/DBDeltaChart.tsx | Adds caller-defined selected expressions and priority-property ordering to shared delta analysis. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
SDK[LLM SDK telemetry] --> CH[(ClickHouse attributes)]
CH --> Normalize[Read-time dialect normalization]
Normalize --> Overview[Overview and attribution]
Normalize --> Latency[Latency and delta analysis]
Normalize --> Sessions[Session timeline]
Sessions --> Detail[Lazy span detail and chat view]
Normalize --> Errors[Trace and log errors]
Reviews (18): Last reviewed commit: "fix(app): cap key-path indices when reco..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @@ -0,0 +1,316 @@ | |||
| import { useCallback, useEffect, useState } from 'react'; | |||
Contributor
There was a problem hiding this comment.
This new component is 316 lines, while LLMSessionPanel.tsx is also 302 lines. Both exceed the repository's 300-line maximum, increasing maintenance cost; split them into smaller focused components.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Comment on lines
+77
to
+81
| !isLoading && ( | ||
| <Text size="sm" c="dimmed"> | ||
| No LLM messages found on this span. Prompt and completion capture | ||
| may be disabled in the instrumentation. | ||
| </Text> |
Contributor
There was a problem hiding this comment.
This no-message branch and the no-session-results branch in LLMSessionPanel.tsx render plain Text elements instead of the required shared EmptyState, bypassing the repository's consistent empty-state presentation and behavior. Use @/components/EmptyState for both branches.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Contributor
E2E Test Results✅ All tests passed • 324 passed • 1 skipped • 1396s
Tests ran across 4 shards in parallel. |
Contributor
Deep ReviewScope: PR #2990 — LLM observability dashboard, span chat view, and sessions (~5,600 production lines, TypeScript/React + query-time ClickHouse SQL generation, base Intent: Add read-time, schema-agnostic LLM observability — a Security review found no injection, authz, or secret-leak issues: user-controlled inputs ( ✅ No critical issues found. 🟡 P2 — recommended
🔵 P3 nitpicks (9)
Pre-existing (not counted toward verdict)
Reviewers (11): correctness, security, reliability, performance, kieran-typescript, frontend-races, testing, maintainability, project-standards, previous-comments, api-contract. (The adversarial reviewer timed out; its focus areas — double-counting election, session-id collisions, cache-hit-rate, and cost/error-rate division — were verified directly during synthesis.) Testing gaps:
|
wrn14897
changed the title
…ion lookup hardening Post-review fixes for the LLM observability branch, driven by a live comparison against opencode's self-reported session cost: the /llm dashboard showed ~$22.72 for a session opencode itself priced at $9.16. Cost accuracy (verified exact against opencode's cost_usd on 111 calls): - Provided-cost election: apps that stamp their own per-call cost (cost_usd / llm.cost.total / gen_ai.usage.cost) are treated as the authoritative reporters, and all token/cost/call aggregations sum only those rows when any exist in scope (llmGatedSumExpr / llmGatedCountExpr, rendered as raw select aggregates). This dedupes dual-instrumented apps — opencode emits OpenInference spans (with cost) AND Vercel AI SDK spans (with gen_ai.usage.*) for every call, in separate traces, so no row-local gate could catch it. - Cache-aware estimation: the SQL cost expression now prices uncached input, cache reads (discounted), cache writes (Anthropic's 1.25x premium, new catalog rate + attribute keys incl. OpenInference prompt_details.cache_write, Vercel inputTokenDetails.cacheWriteTokens, and flat cache_creation_tokens), and output separately, matching the TS-side computeCostUsd. The inclusive/exclusive input-token heuristic now accounts for writes, and totalTokens reports effective context. - Query-size guard: the enriched cost expression embeds the price catalog per token term; the first live run exceeded ClickHouse's 256 KiB max_query_size. Rates are now factored into per-term multiIfs and the whole expression is bound once per query as a WITH expression alias (LLM_COST_SQL_ALIAS), keeping dashboard queries at ~80 KiB. Session drawer correctness (review feedback): - The per-span attribute lookup now pins TraceId alongside SpanId + timestamp (span ids can be empty or collide across traces) and is bounded to the searched window for partition pruning. Also trims the llm module's public surface to what external consumers import (fixes 23 knip unused-export findings) and replaces the unused zod chat-message schemas with plain interfaces. Adds the missing changeset for the LLM observability feature.
The LLM observability branch added 6 eslint-disable comments, tripping the app/eslint-disable ratchet (150 > baseline 144). Remove the escapes by fixing the underlying patterns instead of suppressing them: - Chat messages get a stable `id` assigned in extractConversation (conversations are immutable once extracted), so message lists key on data instead of array indexes. - Session timeline rows carry their accordion `itemValue` in the row data built per fetch, replacing the index-derived key/value pair. - The session filter now lives in the URL only: SessionSelectControlled becomes a plain value/onChange SessionSelect wired straight to the nuqs param, deleting the two deliberately-under-depped form<->URL sync effects (the drawer's "Filter dashboard" action writes the same param). - The trace-source default adoption effect gets full dependencies — the select only offers usable trace sources, so a user selection always resolves to itself and the effect can never fight it.
Adds the standard beta badge (matching Service Map's nav badge) next to the LLM breadcrumb, plus an info hover card explaining what to expect: the dashboard is experimental, which instrumentations it understands, that costs are catalog estimates unless the instrumentation reports its own cost, and how dual-instrumented apps are counted.
Hide the passive LLM surfaces on event/trace views while the feature bakes, mirroring the alert-details flag pattern: default off, enabled in dev via .env.development, opt-in comment in docker-compose.yml. Gated under IS_LLM_PANELS_ENABLED: - the LLM tab in the event row side panel - the LLM section on the row Overview tab - the LLM tab in the trace span detail panel - the model + token-count suffix on waterfall span labels The /llm dashboard and its dashboards-list entry are intentionally not gated — the dashboard is opt-in by navigation and already labeled beta.
Drop all passive LLM surfaces on existing event/trace views so this PR ships only the opt-in dashboard; the side-panel integrations can return in a follow-up: - Restore DBRowSidePanel(+types), DBRowOverviewPanel, DBTracePanel, and DBTraceWaterfallChart to main (removes the LLM tab, overview section, span-detail tab, and waterfall label suffix). - Remove the now-moot IS_LLM_PANELS_ENABLED flag and its env/compose wiring. - Delete the pieces those surfaces orphaned: LLMConversationPanel, lib/rowData (row-data extraction glue), asLLMEvents, their tests, and the @/llm barrels (dashboard code imports defining modules directly). - Reword the changeset to describe the dashboard only. Outside src/llm the branch now only touches AppNav (dashboards-group active state for /llm) and the dashboards-list preset entry.
- Recognize current-registry dotted semconv usage keys (gen_ai.usage.cache_read.input_tokens, gen_ai.usage.reasoning.output_tokens) emitted by GitHub Copilot Chat and newer SDKs, plus copilot_chat.time_to_first_token (ms) for TTFT. - Expand the model price catalog with xAI, DeepSeek, Mistral, Cohere, Qwen, Meta Llama, and Amazon Nova families across bare, OpenRouter, HuggingFace, and Bedrock id flavors. - Add agent attribution (gen_ai.agent.name / agent.name): per-agent calls, tokens, est. cost, and error rate charts on the dashboard and an agent badge in the span subpanel.
…pling The delta breakdown's sampling queries are row-capped (LIMIT 1000 + stable hash order + PartIds indexHint, inherited from search's delta mode), but they SELECT * — and agent SDKs stamp full conversation histories on every span, so a 1000-row LLM-span sample measured ~412 MiB of SpanAttributes (~800 MiB per selection across the outlier + inlier queries). DBDeltaChart gains an opt-in selectExpression (default '*', threaded through DBSearchHeatmapChart as deltaSelectExpression). The LLM Latency tab passes a trimmed select that drops attribute values longer than 256 chars server-side via mapFilter (~500x smaller: 0.11 MiB vs 55 MiB measured on the same sample) — long values are exactly what the breakdown hides as high-cardinality anyway. Falls back to SELECT * for JSON-typed or derived attribute columns, and holds queries until the JSON-column lookup resolves. Search page behavior unchanged.
brandon-pereira
left a comment
brandon-pereira
left a comment
Member
There was a problem hiding this comment.
Overall works well!
You can use the AI Assistant on chart explorer to test this. As we rollout AI in HyperDX users can use this dashboard to observe hyperdx :)
Row hrefs were computed from a window.location snapshot at render time. Cached table rows can render before an async nuqs URL write (e.g. a tab switch) lands, baking the previous tab into the link — clicking a session then navigated back to the old tab underneath the drawer. Build hrefs from useSearchParams() so they recompute when the URL changes, and extract a pure helper with regression tests.
Scope both row tiles to failures: trace spans with an error status and correlated error-severity log events. The top-bar where/session/time scoping still applies, so the tab remains a filterable row browser. tab=search URLs stay accepted and normalize to the Errors tab. Adds a log-side isError expression mirroring the serviceDashboard convention.
Add llmTelemetry(), which builds Vercel AI SDK experimental_telemetry
settings from metadata the SDK flattens onto every span as
ai.telemetry.metadata.* — the keys the /llm dashboard reads for session
grouping (sessionId) and user attribution (userId). Wire teamId/userId
into the chart assistant call.
EE note: the notebook investigation agent should adopt this at its
generateText call site with sessionId set to the notebook id so its
calls group on the LLM dashboard Sessions tab:
experimental_telemetry: llmTelemetry({ sessionId: notebookId, teamId, userId })
Double backslashes before escaping quotes when embedding catalog regex
patterns in string literals: ClickHouse decodes recognized escapes
(\b, \t, \xHH) inside literals, so a future pattern using e.g. a word
boundary would silently corrupt. Write the finish-reason char class
backslash-free ('[]["]'). Adds a round-trip regression test that
simulates ClickHouse literal decoding and re-matches catalog patterns.
Cost display (KPI tiles, Est. Cost columns/charts, drawer and subpanel cost) is now gated on IS_LLM_COST_ENABLED (NEXT_PUBLIC_IS_LLM_COST_ENABLED, default off): the bundled price catalog goes stale between releases and can't account for provider discounts, so tokens are the primary metric until there's a price-management story. The catalog, cost expressions, and provided-cost election stay — the election also dedups token counts, and flipping the flag restores the full cost UX. With the flag off the ~70 KiB cost WITH-binding is skipped on every dashboard query.
The election gated whole aggregation scopes: one provided-cost span anywhere in a KPI tile, time bucket, or mixed group made the gated sums/counts keep only provided-cost rows, silently dropping every token-only app sharing the trace source. Elect per service instead — dual-dialect emission (opencode's OpenInference+cost span plus Vercel spans per call) is an app-level property and both dialects share a ServiceName — via sumMap-keyed per-service branches summed with arraySum. Validated against live ClickHouse: mixed two-service scope now returns all calls/tokens (was dropping the token-only app), and single-service totals are unchanged (gated cost still exactly matches opencode's self-reported cost_usd sum, 250/250 calls).
The drawer body rendered blank while the span-list query was in flight (the empty state was gated on !isLoading but nothing rendered in its place); show a centered loader instead. Header token/cost totals now render only once the totals query returns rather than flashing "0 tokens".
keyPathsToArray accepted any non-negative integer index, so an ingested attribute key like llm.input_messages.2000000000.message.content created a ~2e9-length sparse array — the trailing filter() iterates the whole range (~27s measured in V8), freezing the tab when the span is expanded, and nested arrays hit the same via downstream map(). Reject keys carrying any integer segment above 4096; real conversations top out at a few hundred entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear
https://linear.app/clickhouse/issue/HDX-5162/llm-observability-span-chat-view-cost-tracking-sessions-and-llm
Why
Teams running LLM apps and coding agents (OpenAI/Anthropic SDKs, Vercel AI SDK, LangChain via OpenLLMetry/OpenInference, opencode, Claude Code, GitHub Copilot Chat) already send their telemetry to HyperDX — but the product had zero LLM awareness: no chat rendering, no token/cost concepts, no model analytics. This PR adds LLM observability at feature parity with dedicated LLM observability tools while staying HyperDX-native.
Approach: read-time, schema-agnostic
Unlike dedicated LLM observability tools that rely on ingest-time processing and dedicated tables, everything here derives from span/log attribute maps at query time:
Mapand JSON-typed attribute columns via the source'seventAttributesExpressionWhat's included
Normalization lib (
packages/app/src/llm/lib) — pure, unit-tested TS:gen_ai.usage.cache_read.input_tokens), OpenLLMetry, OpenInference, Vercel AI SDK — plus real-world variants captured as fixtures from opencode, Claude Code, and GitHub Copilot Chat telemetry (whole-stringllm.input_messages, camelCaseai.usage.*, flatinput_tokens/cost_usdkeys, bracketed model ids likeclaude-opus-5[1m],copilot_chat.time_to_first_token)Cost estimation — built but hidden by default (
lib/modelPrices.ts,lib/cost.ts, gated onNEXT_PUBLIC_IS_LLM_COST_ENABLED):gen_ai.usage.cost,llm.cost.total,cost_usd) that always wins over the catalog, and a SQLmultiIfgenerator bound once per query as aWITHalias to stay under ClickHouse'smax_query_size. The provided-cost election also dedups token counts, so it runs regardless of the flag/llmpreset dashboard (Overview | Latency | Sessions | Errors), listed on the Dashboards page:gen_ai.agent.name): per-agent calls, tokens, est. cost, and error rategen_ai.conversation.id→session.id→ai.telemetry.metadata.sessionId) — the correlation surface for instrumentations that stamp session ids but don't propagate trace context. Row click opens a timeline drawer; each call expands into a normalized chat view with role badges, markdown, collapsible tool calls, and a usage/cost summarytab=searchURLs normalize here)API attribution helper (
packages/api):llmTelemetry()builds Vercel AI SDKexperimental_telemetrysettings whose metadata the SDK flattens onto every span asai.telemetry.metadata.*— the keys this dashboard reads for session grouping (sessionId) and user attribution (userId). The chart assistant now stampsteamId/userId; conversation-scoped callers should also passsessionIdso their calls group on the Sessions tabCorrectness & performance notes
gen_ai.usage.*/llm.token_count.*/ flat primary-reporter keys) so SDK wrapper spans (e.g. Vercel'sai.streamTextarounddoStream) don't double count. Dual-dialect apps (opencode emits an OpenInference span withcost_usdand Vercel spans per call) are deduped by a per-service provided-cost election: a service's provided-cost rows win over its other usage rows, while services without provided costs keep all their usage rows — so cost-reporting apps never evict token-only apps from shared aggregatesstopvs["stop"])SELECT *sample measured ~412 MiB — attribute values over 256 chars are dropped server-side viamapFilter(~500× smaller), which loses nothing since long values are hidden as high-cardinality anywayKnown limitations
NEXT_PUBLIC_IS_LLM_COST_ENABLEDto opt inTesting
src/llm/__tests__(per-dialect fixtures lifted from real opencode/Claude Code/Copilot telemetry, cost math and price-catalog matching, SQL expression generation, lazy-loading regression guards)tsc --noEmitclean; eslint 0 errors; knip cleanScreenshots