Dual-emit legacy metric names behind --otel-use-legacy-metrics

[glageju]

Summary

Stacked on #5956. Base is gautam/metrics-std-toolhive, so this diff shows only the dual-emission layer. Review #5956 first; merge it first.

#5956 renames ToolHive's metrics to the stacklok.* vocabulary. @aponcedeleonch and @ChrisJBurns both blocked it on the same point: a rename empties a dashboard panel exactly the way a deletion does, and the renamed metrics have no overlap window — the old name stops and the new one starts in the same release, so there is no moment where a user can run both queries side by side and confirm they agree.

Span attributes already have machinery for exactly this: dual emission behind --otel-use-legacy-attributes, with #6072 tracking its retirement. Metrics went with a hard cutover instead, and the asymmetry wasn't justified. This PR closes it — metrics get the same treatment: a flag, a window, and a documented path.

After this PR, upgrading #5956 breaks nothing by default. All 25 pre-rename metric names are still emitted, under their original label keys.

• Adds the dual-emit seam (pkg/telemetry/legacymetrics.go): helpers that return a working instrument when enabled and a no-op otherwise, so the ~25 record sites stay branch-free instead of growing an if each. An instrument-creation failure also yields a no-op — a legacy alias must never be the reason a process fails to start.
• Threads UseLegacyMetrics end to end — CLI flag, config file, operator CRD, thv serve, and vMCP — defaulting to true at every layer.
• Emits legacy aliases for all 25 old names across proxy, rate-limit, and vMCP metrics, including the six §3.5 semconv twins that Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins #5956 deletes outright. Those cost nothing new to keep: they already coexist with their semconv replacements today.
• Aliases keep their original label keys. This matters more than it looks: the rate-limit metrics change both name and label (toolhive_rate_limit_decisions{server=…} → stacklok_toolhive_ratelimit_decisions_total{mcp_server=…}), so those dashboards break twice over. An alias emitting the old name with the new label key would be a second breaking change, not a compatibility path — so the legacy attribute sets are built separately rather than shared.
• Un-merges the outcome-labelled counters. Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins #5956 folds four _errors/_not_found counters into an outcome label (RFC D4). The aliases fan back out to the original separate counters, and find_tool deliberately keeps its legacy counter incrementing before the call — the original counted attempts, not completions, so it's reproduced as-was rather than silently changing meaning during the overlap window.
• Reframes the migration guide from a one-shot cutover into a deprecation schedule, with the old → new mapping and the removal plan.

Type of change

• New feature

Test plan

• Unit tests (task test) — 199 packages, race detector on
• Linting (task lint-fix) — 0 issues
• Manual testing (described below)

Verified against a running proxy with --otel-enable-prometheus-metrics-path:

• Default (flag on) — both name families present in /metrics.
• --otel-use-legacy-metrics=false — zero ^toolhive_ series; new names still present.
• Label compatibility — toolhive_rate_limit_decisions_total still carries server="…", not mcp_server="…". This is the check that would catch the "breaks twice over" regression, and an unanchored assertion can't catch it: server="x" is a substring of mcp_server="x".
• Un-merge — a not-found call_tool increments both the new outcome="not_found" series and the legacy _not_found counter.

API Compatibility

• This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

Adds one optional field, MCPTelemetryOTelConfig.UseLegacyMetrics, with +kubebuilder:default=true. Purely additive — no field is removed or retyped, and existing CRs are unaffected.

Does this introduce a user-facing change?

Yes, and it makes #5956's change non-breaking by default.

New flag --otel-use-legacy-metrics (also use-legacy-metrics in config, spec.otel.useLegacyMetrics on the CRD), defaulting to true. While on, ToolHive emits both the new stacklok.* metric names and the pre-rename toolhive_* names with their original label keys, so existing dashboards and alerts keep working across the upgrade.

Migrate at your own pace: run the old and new queries side by side, confirm they agree, then set the flag to false to drop the legacy series. The flag and the legacy names will be removed in a later minor — see docs/telemetry-migration-guide.md for the old → new mapping and the schedule.

Dual emission roughly doubles series count for the metrics involved while enabled; histograms are the expensive case.

Changes

File	Change
pkg/telemetry/legacymetrics.go	New. The three dual-emit helpers
pkg/telemetry/config.go	Config.UseLegacyMetrics, default, (*Provider).UseLegacyMetrics() accessor
pkg/telemetry/middleware.go	Proxy legacy aliases; startup deprecation warning
pkg/telemetry/serve.go	Flag default on thv serve
cmd/thv/app/run_flags.go	--otel-use-legacy-metrics; resolveOptionalBool helper
pkg/config/config.go	OpenTelemetryConfig.UseLegacyMetrics
pkg/runner/{telemetry_config,config_builder,middleware}.go	Threading; nil → true
pkg/ratelimit/{observability,limiter,middleware}.go	Rate-limit aliases keeping the server label
pkg/vmcp/core/core_telemetry.go	Composite-tool aliases
pkg/vmcp/internal/backendtelemetry/backendtelemetry.go	Backend aliases
pkg/vmcp/server/sessionmanager/factory.go	9 optimizer aliases; outcome un-merge
pkg/vmcp/{cli/serve,ratelimit/factory/limiter}.go	vMCP threading
cmd/thv-operator/**, deploy/charts/**, docs/{cli,operator,server}/**	CRD field + regenerated
docs/telemetry-migration-guide.md, docs/observability.md	Deprecation schedule

Special notes for reviewers

On RFC D13 ("no dual-emit window"). I don't think this PR contradicts it, for two reasons worth checking rather than taking on faith:

1. D13 scopes itself to "the rename to stacklok.* and the deletion of the six legacy twins", and its supporting argument is entirely about those twins not being scraped in a default shipped deployment. That argument covers the 6 deletions. It doesn't reach the 15 renames, which are the larger share.
2. D13's own closing sentence pre-authorizes this: "If a scraped consumer is identified before the rename lands, revisit this decision for that surface." This repo ships four Grafana dashboards under examples/otel/grafana-dashboards/, which is at minimum evidence of intended scraped consumption.

@reyortiz3 — as RFC author, could you confirm whether D13's scope was meant to cover the renames or only the twins? One sentence settles whether this needs an RFC amendment or just a note. I've deliberately not edited the RFC.

Deliberate design choices, in case they look wrong at a glance:

• Aliases carry the old label keys, not the new ones. Looks like an inconsistency in the diff; it's the whole point. See the rate-limit case above.
• find_tool's legacy counter increments before the call. Also looks inconsistent next to the new outcome counter, which increments after. The original counted attempts, so reproducing it faithfully means keeping that placement.
• legacymetrics.go takes the metric name as a parameter and contains no name literals, so the seam has no opinion about what's being aliased.
• token_savings is the one row where the rename is nearly invisible. Both the new metric and its alias carry unit %, so they export as stacklok_vmcp_optimizer_token_savings_percent and toolhive_vmcp_optimizer_token_savings_percent — the new Prometheus name differs from the old only in prefix, and the exporter does not double the _percent suffix on the alias (verified against a real scrape). Worth a sanity check by a second pair of eyes.

Extra scrutiny welcome on two things:

1. The removal release is unnamed. The guide's deprecation timeline currently says "current release / future release / later release" rather than literal versions, matching how useLegacyAttributes is already documented. Two things are therefore still open, and I'd rather settle the version before shipping either: pinning actual release numbers in the timeline, and the startup deprecation warning @aponcedeleonch and @ChrisJBurns both asked for. A warning that says "a future release" gives users nothing to plan against, so I left it out pending a number. Happy to add both here.
2. Whether the six §3.5 twins should also be retained. This PR keeps them, per the review. If you'd rather Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins #5956's deletion stand and only the 15 renames get aliases, that's a smaller diff and I can cut it back.

Known gap, not fixed here (pre-existing, and orthogonal to dual emission): stacklok.build_info exports as stacklok_build_info_ratio, because coremetrics.RegisterBuildInfo sets metric.WithUnit("1") and the Prometheus translator appends a unit suffix. _ratio is wrong for an identity gauge, and it means a dashboard joining on stacklok_build_info returns empty. The durable fix is upstream in toolhive-core. Happy to file it, or fold it in here if you'd rather it not reach customer dashboards at all.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 90.17094% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.69%. Comparing base (3f960e6) to head (56f55e0).

Files with missing lines	Patch %	Lines
pkg/vmcp/cli/serve.go	0.00%	8 Missing ⚠️
pkg/telemetry/legacymetrics.go	71.42%	3 Missing and 3 partials ⚠️
pkg/runner/middleware.go	77.77%	1 Missing and 1 partial ⚠️
pkg/telemetry/middleware.go	96.82%	1 Missing and 1 partial ⚠️
pkg/telemetry/serve.go	50.00%	2 Missing ⚠️
pkg/vmcp/server/sessionmanager/factory.go	96.00%	2 Missing ⚠️
pkg/telemetry/config.go	83.33%	1 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           gautam/metrics-std-toolhive    #6168      +/-   ##
===============================================================
+ Coverage                        72.68%   72.69%   +0.01%     
===============================================================
  Files                              736      737       +1     
  Lines                            76447    76650     +203     
===============================================================
+ Hits                             55563    55724     +161     
- Misses                           16951    16985      +34     
- Partials                          3933     3941       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.