Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins

[glageju]

Summary

The proxy and vMCP used toolhive_mcp_* / toolhive_vmcp_* metric names, emitted six legacy metrics alongside their emerging-semconv replacements, and diverged from the platform label vocabulary. This finishes the semconv migration and adopts the shared toolhive-core vocabulary, per the Metrics Standardization RFC.

• Proxy (pkg/telemetry/): toolhive_mcp_active_connections → stacklok.toolhive.proxy.active_connections; build_info via coremetrics.RegisterBuildInfo; added http.server.request.duration (OTel HTTP semconv) so deleting the legacy MCP twins doesn't drop transport-level coverage; D8 ownership labels (stacklok_component=toolhive, stacklok_product=stacklok-platform) promoted to every series via WithResourceAsConstantLabels and hardened as the final OTEL resource detector so a CLI custom attribute or OTEL_RESOURCE_ATTRIBUTES entry for those keys cannot override the frozen values.
• vMCP (pkg/vmcp/): toolhive_vmcp_workflow_* → stacklok.vmcp.composite_tool.*; backend mcp_server.health gauge rename to a live per-(mcp_server, state) gauge, keyed by workload ID for consistency with the backend registry and health-status provider; optimizer renames + token_savings.
• ⚠️ Breaking: six legacy semconv twins deleted (toolhive_mcp_requests, toolhive_mcp_request_duration, toolhive_mcp_tool_calls, toolhive_vmcp_backend_requests, toolhive_vmcp_backend_errors, toolhive_vmcp_backend_requests_duration). Their semconv replacements (mcp.server.operation.duration, mcp.client.operation.duration, http.server.request.duration) are kept.
• Bumps the toolhive-core dependency to v0.0.34, the release carrying telemetry/metrics (adopts core label keys, outcome values, and bucket presets). The transitive AWS-SDK and grpc bumps in go.mod come from toolhive-core's own requirements.
• Updates integration and e2e tests (test/integration/vmcp/, test/e2e/) that asserted the deleted/renamed metric names and old label scheme (mcp_method→mcp_method_name, server→mcp_server, status→HTTP status code) to assert the semconv replacements.
• Updates the Grafana example dashboards and docs/telemetry-migration-guide.md for the renamed/deleted metrics, including a complete old→new metric and label mapping and a corrected MCP Request Rate vs. total HTTP request-rate query split.
• Enables the Grafana dashboard sidecar in the example otel dev stack (examples/otel/prometheus-stack-values.yaml) so the updated example dashboards can be auto-provisioned into a local Kind cluster via a labeled ConfigMap, for validating the renamed metrics land correctly end-to-end.
• Extends the rename to pkg/ratelimit/observability.go, missed by the original pass: toolhive_rate_limit_decisions/_redis_errors/_check_latency → stacklok.toolhive.ratelimit.{decisions,redis_errors,check_latency}, with the migration guide and docs/observability.md updated to match.
• Renames toolhive_vmcp_backend_revision_reclassifications (added independently on main after this branch forked) to stacklok.vmcp.backend.revision_reclassifications so it doesn't reintroduce the legacy naming scheme this PR retires.
• Clarifies docs/telemetry-migration-guide.md's Backward Compatibility section: span attributes and metrics follow different policies (dual-emission vs. hard cutover) for different reasons — the guide previously implied one uniform strategy.

Type of change

• New feature

Test plan

• Unit tests (task test)
• Linting (task lint-fix)
• Manual testing (describe below)

Verified the shipped standalone path (GOWORK=off, published toolhive-core v0.0.34, no workspace):

• GOWORK=off go build ./... — clean.
• GOWORK=off go test ./pkg/telemetry/... ./pkg/vmcp/... — all packages pass, including the D8 label-promotion assertion in pkg/telemetry/buildinfo_test.go (stacklok_component="toolhive" / stacklok_product="stacklok-platform" promoted to per-series labels) and the updated TestVMCPServer_Telemetry_CompositeToolMetrics integration test.
• task test (full suite) is green.
• The e2e telemetry tests (test/e2e/) were updated for the new metric/label vocabulary and compile clean (go vet); they require a full task test-e2e run (containers) to exercise end-to-end.

API Compatibility

• This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

Does this introduce a user-facing change?

Yes. Prometheus metric names emitted by the proxy, vMCP, and rate limiting change to the stacklok.* namespace, and six legacy toolhive_* twins are removed. Their OTel-semconv replacements cover the same signals; docs/telemetry-migration-guide.md maps old → new. A customer scraping the old names loses those series (RFC D13 accepts this: the deleted twins are not scraped in a default shipped deployment).

Special notes for reviewers

• Depends on toolhive-core v0.0.34 (the shared-vocabulary release); this PR's go.mod pins it.
• toolhive-core v0.0.34 trims the Component* value roster a pre-release build referenced; toolhive defines its own stacklok.component = "toolhive" constant (pkg/telemetry/middleware.go, pkg/telemetry/providers/providers.go), asserted by pkg/telemetry/buildinfo_test.go.
• pkg/vmcp/internal/backendtelemetry/backendtelemetry.go's health gauge resolves each backend's status with a three-tier precedence — a live health.StatusProvider first, then request-outcome tracking, then the registry's discovery-time snapshot — matching the same precedence filterHealthyBackends uses for capability aggregation. core.New unregisters the gauge's callback on every construction-failure path after registration succeeds, and on Close(), so a partially-constructed coreVMCP never leaks a callback against the shared meter provider.

Links

• RFC: Stacklok Platform Metrics Standardization (D1, D2, D5, D6, D8, D11; §3.5 twin deletion).
• Telemetry migration guide — old→new metric/label mapping and upgrade steps.
• Telemetry: retire legacy span attribute dual-emission (useLegacyAttributes) #6072 — follow-up to retire the separate useLegacyAttributes span-attribute dual-emission flag (out of scope here; metrics/labels in this PR use a hard cutover with no such flag).

Ref: https://github.com/stacklok/stacklok-enterprise-platform/issues/1380
Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 92.25352% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.64%. Comparing base (a48448d) to head (d4e44a2).

Files with missing lines	Patch %	Lines
pkg/telemetry/middleware.go	84.14%	9 Missing and 4 partials ⚠️
pkg/vmcp/core/core_vmcp.go	87.87%	2 Missing and 2 partials ⚠️
...vmcp/internal/backendtelemetry/backendtelemetry.go	96.42%	3 Missing ⚠️
pkg/telemetry/providers/providers.go	90.90%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5956      +/-   ##
==========================================
+ Coverage   72.62%   72.64%   +0.01%     
==========================================
  Files         736      736              
  Lines       76359    76447      +88     
==========================================
+ Hits        55457    55533      +76     
- Misses      16962    16997      +35     
+ Partials     3940     3917      -23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[glageju]

Addressed the seven inline findings in 46864d8: mcp_server label restored on the two proxy request metrics, SSE connection duration split into its own histogram, the OwnershipLabels escaping split fixed, migration guide mapping table corrected, and the two small doc/comment issues cleaned up.

On the cutover strategy: I don't have a counter-argument to D13 only covering the 6 deletions, not the 18 renames. Dual-emitting the renamed metrics behind a flag defaulting to on, keeping the 6 deleted twins one more release, and adding a startup WARN is a reasonable path and consistent with how span attributes are already handled via --otel-use-legacy-attributes. That's a design change beyond what I can fold into responses to this review's inline comments, so I'm not implementing it here. Will follow up on scoping that as a separate change.

[ChrisJBurns]

Cutover strategy: remove legacy attributes, but keep renamed metrics behind a flag

Seconding @aponcedeleonch's request — this is the one thing I'd want resolved before merge, and I want to add some evidence for why the rename half is the more dangerous half.

The position: retiring the legacy span attributes is fine. That surface already has a compatibility story — dual emission behind --otel-use-legacy-attributes, with #6072 tracking its retirement. Users get a flag, a window, and a documented path.

Renaming metrics has none of that today, and it should get the same treatment:

• Dual-emit the renamed metrics under both old and new names, behind an --otel-legacy-metrics flag defaulting to on.
• Keep the 6 deleted twins one more release. They already coexist with their semconv replacements today, so keeping them costs nothing new.
• Emit a startup WARN naming each deprecated metric and the release it will be removed in.
• Document the deprecation — an explicit old → new table plus a stated removal release. The mapping table in this PR is already most of the way there; it needs to be framed as a deprecation schedule rather than a one-shot cutover.
• Remove both in a later minor, and call it out prominently in the release notes.

Why the renames are worse than the deletions

I verified the counts against the mapping table at a8c3103b: 21 legacy toolhive_* metric names — 6 deleted, 15 renamed — plus 3 counters merged into outcome labels. After this PR, toolhive_ appears in zero metric names in non-test Go source.

The intuition here inverts, and I think it's the crux of the disagreement:

• The 6 deletions are the least disruptive category. Their semconv replacements (mcp.server.operation.duration, mcp.client.operation.duration) already exist and are already emitted today, alongside the twins. A user can migrate their queries before upgrading, run both, confirm they agree, then upgrade. That overlap window is exactly what RFC D13's argument rests on.
• The 15 renames + 3 merges have no overlap window at all. The old name stops and the new name starts in the same release. There is no moment where a user can run both and compare.

So D13 justifies the category that was already safe, and doesn't reach the category that isn't. A rename empties a dashboard panel exactly the same way a deletion does.

It also compounds: on the rate-limit metrics both the metric and its label change (toolhive_rate_limit_decisions{server=…} → stacklok_toolhive_ratelimit_decisions_total{mcp_server=…}), so those dashboards break twice over.

Three breakages not currently in the guide

1. Histogram bucket boundaries change on metrics whose names survive. mcp.server.operation.duration moves from BucketsMCPSemconv() […0.02, 0.2, 2…] to BucketsMCPProxy() […0.25, 2.5…]. Any histogram_quantile() spanning the upgrade mixes two incompatible layouts and returns silently wrong numbers. This is also the unresolved #6144 conflict — see the inline comment.
2. telemetry.MCPHistogramBuckets is deleted from a public package. Exported identifier in github.com/stacklok/toolhive/pkg/telemetry, currently referenced from six places in-repo. External importers break at compile time, not just at query time. Not mentioned anywhere in the guide.
3. stacklok.build_info is documented under a name Prometheus never emits. It exports as stacklok_build_info_ratio (the WithUnit("1") → _ratio rule). Reproduced against this repo's pinned deps. So even a user who does the migration work lands on a query matching nothing.

And docs/telemetry-migration-guide.md:213 still tells operators "Your existing dashboards and alerts continue to work without changes" — which contradicts the Backward Compatibility table two sections above it.

One note on risk assessment: green CI isn't much signal here. The repo ships no PrometheusRule or alert artifacts, so nothing in-repo can catch a bad metrics migration. The first signal is a customer's dashboard going blank.

---

Review findings

Reviewed at a8c3103b across six specialist passes (OTel semconv, Go concurrency/lifecycle, backwards-compat/docs, test coverage, security/config, architecture). Claims marked verified were reproduced against this repo's pinned dependencies.

Already fixed in 46864d8e — re-verified, no action needed: mcp_server restored on both proxy request metrics; OwnershipLabels switched to literal Prometheus label names; SSE split into its own stacklok.toolhive.proxy.sse_connection.duration on BucketsMCPProxy(). Good fixes.

Consensus summary

Severity	Finding	Location
HIGH	http.request.method recorded raw — unbounded attacker-controlled cardinality; semconv _OTHER normalization missing	middleware.go:747
HIGH	Semconv histogram buckets switched to the coarser proxy preset — reverts #6144, breaks quantile continuity	middleware.go:86
HIGH	stacklok.build_info exports as stacklok_build_info_ratio; test passes on substring	buildinfo_test.go:41
HIGH	Step 1 still promises existing dashboards keep working	telemetry-migration-guide.md:213
HIGH	Documented cardinality bound doesn't exist — gen_ai.tool.name gets the same unbounded value	observability.md:231
MEDIUM	build_info registered per-middleware, not once per process, with no unregister — doc overclaims	middleware.go:127
MEDIUM	mcp.method.name is an unvalidated JSON-RPC method string from the request body	middleware.go:707
MEDIUM	backendHealth.states never pruned — unbounded map keyed by workload ID	backendtelemetry.go:242
MEDIUM	Health gauge flips to unhealthy on context.Canceled and auth failures	backendtelemetry.go:373
MEDIUM	HealthProviderSetter's two-phase init rests on a premise that isn't true	backendtelemetry.go:203
MEDIUM	Unregister cleanup manually replicated across 5 error paths	core_vmcp.go:163
MEDIUM	Optimizer tool_name is unvalidated client input, now multiplied by outcome	factory.go:418
MEDIUM	Test: "health monitor build fails" case is vacuous; doc comment claims a 5th branch with no case	core_vmcp_test.go:226
MEDIUM	Test: no dp.Count assertion (double-record passes) and no 4xx case for the documented narrowing	middleware_test.go:2555
MEDIUM	Test: tautological guard gates the positive assertions	integration_test.go:181
MEDIUM	Test: e2e helper passes silently on zero regex matches	telemetry_metrics_validation_e2e_test.go:360
MEDIUM	Test: validateNoEmptyLabels has no zero-match guard after narrowing to one metric	telemetry_metrics_validation_e2e_test.go:334
MEDIUM	Test: dropped server-identity assertions without replacement	telemetry_integration_test.go:338
MEDIUM	"What Is New" table missing the three genuinely new metrics	telemetry-migration-guide.md:29

Not anchorable inline (outside the diff)

• examples/otel/grafana-dashboards/toolhive-mcp-otel-semconv-dashboard.json:246 and :974 — the semconv dashboard's own Error Rate panels still use error_type!="", which is exactly the 4xx trap the migration guide warns about. The other three dashboards were correctly moved to http_response_status_code=~"[45].."; these two weren't, so the shipped dashboards now disagree on what "error rate" means and this one silently under-reports 403s and 429s.
• docs/arch/10-virtual-mcp-architecture.md:885 — still references the removed mcp_method label. CLAUDE.md requires docs/arch/ to be updated when architecture changes.
• .claude/agents/site-reliability-engineer.md:23 — still points at the deleted pkg/vmcp/server/telemetry.go.
• Three rewritten dashboards lack sum by (...) — rate(mcp_server_operation_duration_seconds_count{…}) with legendFormat: "{{mcp_method_name}}" and no aggregation fans tools/call out to one series per tool, all rendered with the same legend string. The semconv dashboard does this correctly (sum by (le, mcp_method_name)), so the four are now inconsistent.
• docs/operator/virtualmcpserver-observability.md — the migration guide now points readers here for the optimizer and revision_reclassifications metrics, but this doc documents none of them.

Worth noting on the positive side

The callback lifecycle work holds up under scrutiny: every one of the five post-registration error paths in core.New unregisters, Close() is idempotent (the SDK's Unregister is safe twice), the observable callback is genuinely non-blocking (registry.List is an RLock map copy), and the shared state it touches is race-free. The rename itself is also complete — repo-wide greps turn up no missed producer or consumer in deploy/, charts, ServiceMonitors, or alert rules. And replacing nil instruments with noop.* fallbacks quietly fixes a pre-existing panic path.

Scope

At 751 lines of production change across 9 files this is ~88% over the 400-line budget in CLAUDE.md, and three genuine behaviour changes (the health gauge, http.server.request.duration, the D8 hardening) are interleaved with a rename-dominated diff. Given the PR is already 20+ commits deep I don't think splitting now is worth it — but the backend-health gauge is the piece that would most have benefited from separate review.

🤖 Generated with Claude Code

[ChrisJBurns]

HIGH — http.request.method is recorded raw: unbounded, attacker-controlled cardinality.

Go's net/http accepts any RFC 7230 token as a method, and /metrics plus the MCP port are unauthenticated by default here. curl -X ZZZZ1, -X ZZZZ2, … mints a permanently-retained series per distinct method — each multiplied across the bucket set — in both the SDK's in-memory aggregation and Prometheus.

OTel HTTP semconv mandates exactly the missing mitigation: methods outside the known set MUST be recorded as _OTHER, with the original kept on spans only (http.request.method_original, which should not go on the metric).

The deleted toolhive_mcp_requests carried the same raw method, so this isn't net-new — but it's reintroduced on a metric this PR presents as semconv-compliant, and it's now the one metric recorded on every request path.

Separately, semconv lists url.scheme as Required on http.server.request.duration, and network.protocol.version / server.address as recommended. httpProtocolVersion(r) and mapTransport() are already computed a few lines down for mcp.server.operation.duration, so the data is at hand. http.route should be added only from a bounded route pattern — never the raw r.URL.Path.

[glageju]

Addressed in 09b576f.

[glageju]

Addressed in 09b576fdf.

normalizeHTTPMethod maps anything outside the nine RFC 9110 methods to _OTHER. Verified by hand: five curl -X bogus-N requests collapse to a single _OTHER series.

On http.request.method_original — I didn't add it, and I've corrected the comment that claimed I had (3f960e6b8). The span keeps the raw value under the spec name http.request.method; moving it to _original belongs with the rest of the span-attribute work (RFC D9), so I left the metric bounded and the span as-is rather than half-applying the spec.

Also added the semconv-Required url.scheme and recommended network.protocol.version while here.

[ChrisJBurns]

HIGH — this reverts #6144, and changes bucket edges on an already-shipped metric.

247480f16 ("Adopt core MCP semconv bucket preset in telemetry", #6144) landed on main and deliberately put this exact line on BucketsMCPSemconv(). This branch moves it to BucketsMCPProxy(). @aponcedeleonch flagged this as the live rebase conflict and it's still unresolved at a8c3103b.

toolhive-core's own doc comment says BucketsMCPSemconv exists "for emitters that promise semconv fidelity" and names mcp.server.operation.duration as the example — so a semconv-named metric should carry semconv buckets.

Beyond the revert, the edges themselves change: [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, …] → [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, …]. Any histogram_quantile() over a range spanning the upgrade mixes two incompatible layouts and returns silently wrong values. The migration guide documents zero bucket changes.

pkg/ratelimit/observability.go:80 and pkg/vmcp/core/core_telemetry.go:57 make the same substitution without conflicting textually and want the same look. On the latter: toolhive-core documents BucketsLongRunning() as being "for long-running measurements such as sync, reconcile, or composite-tool durations" — naming that exact use case.

[glageju]

Addressed in 09b576f.

[glageju]

Addressed in 09b576fdf. mcp.server.operation.duration and mcp.client.operation.duration are back on BucketsMCPSemconv(); #6144 stands.

The root cause was a single package-level MCPHistogramBuckets var serving three measurement classes, so one edit moved buckets on unrelated metrics. That var is gone and each call site now names its preset.

Two histograms do change edges deliberately: rate-limit latency → BucketsMCPProxy() and composite-tool duration → BucketsLongRunning() (which toolhive-core documents for exactly that case). Called out rather than left silent.

Your point generalised further than the three lines you flagged: the dual-emit aliases on #6168 had the same problem — six re-emitted their original name on BucketsMCPProxy() when those names shipped on semconv. Fixed there in 4565242c8, with a test pinning the boundaries.

[ChrisJBurns]

HIGH — this assertion cannot fail for the property it claims to check, and the documented metric name doesn't exist.

coremetrics.RegisterBuildInfo sets metric.WithUnit("1"), and the Prometheus translator appends _ratio to any gauge with unit "1". I reproduced this against this repo's pinned deps:

stacklok_build_info_ratio{commit="abc123",component="toolhive",version="v1.2.3"} 1

require.Contains(t, body, "stacklok_build_info") is satisfied by stacklok_build_info_ratio as a substring, so the test passes while docs/observability.md:271 documents a name that matches nothing. Any fleet dashboard joining on stacklok_build_info returns empty.

Anchor the assertion at line start (^stacklok_build_info(_ratio)?\{) so the exported name is pinned, and reconcile with the docs. The durable fix is upstream — drop WithUnit("1") for a unitless identity gauge — worth filing before this vocabulary reaches customer dashboards.

Also worth asserting the labels: build_info always observes 1, so component="toolhive" / non-empty version / non-empty commit are the entire point of the metric, and none of them is currently checked.

[glageju]

Addressed in 13d69bc4b, 6f3c6c882, and 3f960e6b8. You were right on both halves, and the second was worse than the comment suggests.

The assertion is now anchored at line start ((?m)^stacklok_build_info(_ratio)?\{) and asserts all three labels, since the value is always 1.

On the name: confirmed stacklok_build_info_ratio against a real scrape. docs/observability.md now documents the exported name and flags _ratio as wrong for an identity gauge — the migration guide already had this right, observability.md didn't.

The actual fix can't happen in this repo: RegisterBuildInfo and its WithUnit("1") live in toolhive-core, so it needs a core change plus a pin bump here. Filed as #6170. Until that lands the (_ratio)? in the test regex and the docs note stay; both come out with the pin bump.

Separately, chasing this surfaced a real bug in my own once-guard — see reply on 3685784106.

[ChrisJBurns]

HIGH — this is now false, and it's the first thing an operator reads when deciding whether an upgrade is safe.

Six metrics are deleted, 15 renamed, 3 merged, and several label keys changed. The Backward Compatibility table two sections above explicitly states metrics are a hard cutover, no legacy fallback — this paragraph directly contradicts it.

The dual-emission reassurance is true only for span attributes. Suggest retitling to scope it explicitly, e.g. "Step 1: Upgrade — Span Attributes Need No Action", and stating plainly that metric-based dashboards and alerts do break, with a pointer to Step 2.

Related: Step 4 ("Update Dashboard Panels") still advises running old and new queries "side-by-side during migration to verify equivalence" — impossible for metrics with no dual emission. Worth rescoping to trace-based panels.

[glageju]

Fixed by #6168 rather than by an edit here.

That PR is open and stacked on this branch: --otel-use-legacy-metrics defaults to true and emits all 25 pre-rename names under their original label keys. Once it lands, "Step 1: No Action Required" is true for metrics as well as spans — the state this section already describes.

The guide is reworked there too: the metric half becomes a deprecation schedule rather than a cutover, and the Backward Compatibility table you cite two sections above no longer says "hard cutover, no legacy fallback".

Being explicit about the dependency: if #6168 doesn't land, this section is wrong as written and needs the retitle you suggested plus a plain statement that metric dashboards break. I've left it pending that PR rather than writing it both ways.

[ChrisJBurns]

HIGH — this cardinality claim doesn't hold, and it will mislead operators into thinking the vector is closed.

recordOperationDuration (middleware.go:791) sets gen_ai.tool.name to resourceID — the same parsedMCP.ResourceID value, i.e. params.name straight from the client's JSON-RPC body. For tools/call and prompts/get the unbounded value simply moved to a different key on the same metric; only resources/read (the URI) genuinely lost it.

Since mcp.server.operation.duration is now the sole per-method metric after the twins were deleted, arbitrary params.name values grow the series set without bound.

Either keep gen_ai.* names on spans only, or bound them against the aggregated capability set the proxy already knows (emitting _OTHER for unresolved names). Either way the note needs correcting — as written it asserts a control that isn't there.

[glageju]

You are right that the note asserts a control that is not there, and that the value simply moved keys rather than going away. The doc fix is coming in the docs commit.

On the code fix, I dug into both branches you offered and neither lands cleanly, so I want to flag what I found rather than pick quietly.

"bound them against the aggregated capability set the proxy already knows" — the proxy does not know it. Checked both paths:

• Plain proxy: nothing in the runner/transport path retains a tools/list result. telemetry.CreateMiddleware's only window onto wider state is types.RunnerConfig, which is GetName() + GetPort(). Reaching RunConfig.ToolsFilter needs a type assertion, which .claude/rules/go-style.md calls out explicitly.
• --tools is user-supplied and skipped entirely when empty (pkg/runner/middleware.go:129), so it bounds nothing for most workloads. ToolsOverride also means the wire name is not the backend name.
• vMCP has a real set (RoutingTable.Tools), but only behind core.ListTools(ctx, identity) — async, per-identity, 30s TTL, invalidated by list_changed. Not snapshottable at construction, and using the authorized view would leak whether a caller is authorized for a tool into a metric label.

Worth noting: backendtelemetry.go:396 already keeps gen_ai.tool.name span-only for the outbound leg — its metric attrs are just mcp.method.name / network.transport / mcp_server. So your branch 1 is what the codebase already converged on, and the two surfaces are currently inconsistent.

What I did: left the attributes as-is here. They predate this branch (they were on recordOperationDuration at the merge-base), so removing them drops a working per-tool metric breakdown that is not in the guide's breaking-change list — a user-visible removal that deserves its own review rather than riding along in a rename. Filing a follow-up for the bounding, where the span-only-vs-distinct-value-cap tradeoff can be decided on its own.

Your point that the deletions make this the sole per-method metric is well taken and is the reason it needs fixing at all — just not, I think, in this PR.

[glageju]

Addressed in 4860adcbc — as a corrected claim, not a fix.

You're right that the unbounded value moved keys rather than going away: resourceID is the same params.name, so tools/call and prompts/get still carry it, and only resources/read genuinely lost it. The doc now says so plainly, including that this is the sole per-method metric once the twins are retired, so there's no unaffected fallback.

Not fixed here: bounding it needs the resolved tool/prompt set, which the middleware has no access to — it sees a parsed request, not the backend's capability list. That's a design change, not a guard. Filed as #6169, covering this and the vMCP tool_name case from your other comment.

Interim mitigation is documented: drop the attributes with a Prometheus metric_relabel_config if untrusted clients can reach the proxy.

[ChrisJBurns]

MEDIUM — this guard is a prefix of the string it guards, so the assertion is tautological.

metricActiveConnections == "stacklok_toolhive_proxy_active_connections", and the guard is strings.Contains(metricsBody, "stacklok_toolhive_proxy"). The first assertion can therefore never fail, and the second only runs when the gauge already happens to be present. If the middleware emitted nothing at all, the whole block is skipped and the test still passes.

The test already made a real request through the middleware, so both series must exist — dropping the if and asserting unconditionally is strictly stronger.

(The NotContains assertions above this are unconditional and genuinely valuable — this is only about the guarded positive half.)

[glageju]

Addressed in 70302f0a5. Guard removed, assertions now unconditional.

Removing it exposed exactly what you'd expect it to be hiding: the request-driving subtests ran under t.Parallel(), so they hadn't executed when the parent scraped /metrics — the metric was genuinely absent from the scrape the test claimed to validate. Requests are now served inline before the scrape.

[ChrisJBurns]

MEDIUM — the regex was weakened and the helper still can't fail.

The pattern went from toolhive_mcp_requests_total\{.*server="X".*transport="Y".*\} (\d+) to mcp_server_operation_duration_seconds_count\{[^}]*\} (\d+), dropping both the server-name and transport pinning. The surrounding if len(matches) > 0 { … } guard is left in place, so if the regex matches nothing the entire body — including Expect(totalRequests).To(BeNumerically(">", 0)) — is skipped and the helper passes silently.

Expect(matches).ToNot(BeEmpty(), "expected ... series in scrape") makes it capable of failing. Now that 46864d8e has restored mcp_server on this metric, the server-identity pinning can also come back.

[glageju]

Addressed in 70302f0a5. Expect(matches).ToNot(BeEmpty()) replaces the if len(matches) > 0 guard, and the server-name pinning is back now that 46864d8ee restored mcp_server on that metric.

[ChrisJBurns]

MEDIUM — no zero-match guard after narrowing the scan from many series to one.

This previously scanned every line containing toolhive_mcp; it now scans only stacklok_toolhive_proxy_active_connections. If that single gauge is missing from the scrape for any reason — instrument creation fell back to the new noop.Int64UpDownCounter{}, a rename, an exporter change — the loop body never executes and the helper passes without asserting anything.

Unlike analyzeTraceAttributes, whose caller asserts TracesGenerated > 0 downstream, there's no backstop here. A checkedLines counter plus Expect(checkedLines).To(BeNumerically(">", 0)) closes it.

[glageju]

Addressed in 70302f0a5 — checkedLines counter with Expect(checkedLines).To(BeNumerically(">", 0)).

[ChrisJBurns]

MEDIUM — server-identity assertions were dropped without replacement, in the one test that scrapes a real vMCP /metrics.

The rewrite deleted assert.Contains(metrics, 'server="telemetry-vmcp"'), 'transport="streamable-http"', and 'mcp_resource_id="search-svc_search"', replacing them only with mcp_method_name= checks. So the central server → mcp_server rename this PR performs isn't verified end-to-end anywhere. Same in TestIntegration_TelemetryRunsBeforeClassificationRejection, where server="telemetry-ordering-vmcp" gave way to a status-code check.

mcp_server="telemetry-vmcp" now lands on stacklok_toolhive_proxy_active_connections and mcp_client_operation_duration_seconds, so asserting it is cheap. The transport= assertion is unaffected by this PR and could simply come back.

[glageju]

Addressed in 70302f0a5. mcp_server and transport assertions restored in both tests.

One detail worth recording: the negative form needs a delimiter anchor. NotContains("server=\"x\"") is satisfied by mcp_server="x" as a substring, so it can't fail — it's now NotRegexp(`[{,]server="x"`). Same trap as the guard in 3685784177, different shape.

[ChrisJBurns]

MEDIUM — the "What Is New" table contradicts the "What Changed" section above it.

"What Changed" was updated to describe three new histograms, but this table still lists only mcp.server.operation.duration and mcp.client.operation.duration. Missing:

• http.server.request.duration
• stacklok.toolhive.proxy.sse_connection.duration (added in 46864d8e)
• stacklok.build_info
• stacklok.vmcp.mcp_server.health — semantically new, not a rename of toolhive_vmcp_backends_discovered
• the new outcome label vocabulary

Worth adding here since this table is what a reader scans to find out what they gain in exchange for the queries they have to rewrite.

[glageju]

Addressed in 6cb43cb76.

stacklok.vmcp.mcp_server.health is listed as semantically new rather than a rename of backends_discovered — worth the distinction, since the old one was a startup constant and the new one is a live gauge.

[glageju]

Thanks Chris — this was a genuinely useful review. Two of your comments found bugs whose first fix was wrong in a way I'd have shipped:

• 3685784128 (cancellation): the guard I added matched context.Canceled, but the client severs that chain with %v and wraps only the domain sentinel with %w. It compiled, read correctly, and never fired. Fixed in ca7a252a2 against vmcp.ErrCancelled/ErrTimeout.
• 3685784106 (build_info): my process-wide sync.Once fixed the duplicate callbacks and made every provider after the first export no build_info at all. Fixed in 6f3c6c882, keyed per MeterProvider.

Your bucket comment also generalised further than the lines you flagged — six dual-emit aliases on #6168 had the same problem under unchanged metric names. Fixed there. And fixing two of your comments left a third comment stale, which d4e44a27d cleans up.

Three things resolved outside this PR, with reasons in the individual replies:

• 3685784090 (Step 1 "No Action Required") — Dual-emit legacy metric names behind --otel-use-legacy-metrics #6168, open and stacked on this branch. --otel-use-legacy-metrics defaults on and emits all 25 pre-rename names under their original label keys. That's the compatibility window you and @aponcedeleonch asked for.
• 3685784101 / 3685784156 (unbounded tool/prompt names) — Bound client-controlled tool/prompt name cardinality on per-request metrics #6169. Bounding them needs the resolved capability set, which neither the proxy middleware nor the optimizer layer has in scope. Interim metric_relabel_config mitigation is documented.
• stacklok.build_info exports as stacklok_build_info_ratio (drop WithUnit("1") upstream) #6170 — the build_info _ratio fix, which has to happen in toolhive-core and can't be done from this repo.

One design decision I've made rather than deferred: 3685784135 — keeping HealthProviderSetter. The false justification is fixed; the real reason is that it keeps the gauge's unregister func acquired at one point early in New, and reordering would push more error paths above the registration — the fragility you flagged in 3685784145.

Everything else in your review is addressed on the branch.