feat(guest-agent): expose GPU telemetry via GpuInfo RPC - #1178
Merged
Conversation
GPU telemetry is an attachment, not a SystemInfo field. A stuck NVML call must not share SysInfo's 20s timeout, and VMM's ProxiedGuestApi decodes then re-encodes so a new field on SystemInfo would be dropped unless the proxy is updated in lockstep. Named GpuInfo to match Info / SysInfo / NetworkInfo. Optional scalars distinguish "query failed" from a genuine zero. Fake memory_reserved_bytes / memory_usable_bytes are omitted: nvml-wrapper 0.12.1 has no memory_info_v2. cc_ready is the system-wide NVML CC ready state (nvmlSystemGetConfComputeGpusReadyState via Device.get_confidential_compute_state). cc_enabled is per GPU via Device.is_cc_enabled(). Together they tell "this card has CC on" from "the driver handshake is not done yet".
Nvml::init() is not request-scoped: Drop shuts the library down, and /metrics would otherwise init+shutdown every 15s. A process-level OnceLock keeps one handle; init failure is cached for 60s so images without a driver do not dlopen on every call. Sampling is serialized with try_lock. The RPC 5s timeout does not cancel a stuck NVML call; later callers return the last snapshot or "sampling in progress" instead of queueing behind the hung sample. NotSupported field errors log at debug (CC mode disables some counters by design). Other per-field errors warn once, then debug. XID event health checks are out of scope.
ProxiedGuestApi decodes then re-encodes. A new guest field that VMM does not know is dropped, so the proxy method is added in lockstep.
/metrics and the dashboard each call GpuInfo separately so a GPU failure cannot blank system info. Optional fields are omitted from Prometheus when NVML did not return them. Labels carry index, uuid, and pci_bus_id, matching dcgm-exporter. dstack_gpu_nvml_up distinguishes "no GPU" from "NVML failed". dstack_gpu_query_errors counts failed fields per card.
Leechael
force-pushed
the
feat/guest-agent-gpu-observability
branch
from
September 6, 2026 03:54
eb3389b to
df0e32c
Compare
Leechael
changed the title
Leechael
marked this pull request as ready for review
September 6, 2026 06:21
kvinwang
force-pushed
the
feat/guest-agent-gpu-observability
branch
from
September 6, 2026 07:45
102e883 to
3eb25e3
Compare
Collaborator
|
Ran this branch on real hardware — an H200 SXM 141 GB ( The three unknownsThe set of NVML fields that return One sample takes 94-99 ms, fork to exit, including The zero-cost claim for GPU-less guests holds on real hardware once the PCI gate is in: 0 collector processes, 0 NVIDIA devices on PCI, across repeated scrapes. Incidentals: One thing the live run found
Full nine-check matrix, plus fixes for that and a few smaller things (a GPU-less guest forking a resident helper on every scrape, |
dstack-util's boot attestation gate already counted display-class PCI devices through sysfs. The guest agent needs the same answer to decide whether GPU telemetry is worth collecting at all, and a second copy of the vendor/class matching is how the two drift apart. The two callers want opposite failure policies -- the boot gate must fail closed when the inventory cannot be read, a telemetry gate wants to report no GPUs -- so the shared function returns the counts and each caller keeps its own policy on top.
nvml-wrapper's Device::is_cc_enabled calls nvmlSystemGetConfComputeSettings and never touches the device handle, so cc_enabled was a system-wide setting copied onto every row and described as per-GPU. It moves next to cc_ready, which is system-wide for the same reason. sample_age_ms is added because the agent now serves the last known snapshot rather than nothing when a sample is overdue. A consumer that cannot see the age cannot decide whether the numbers still mean anything. GpuDevice.error becomes repeated errors. It was a "; "-joined string that the metrics template split back apart to count failures, which any NVML message containing that separator would inflate.
NVML calls cannot be cancelled, so the sample belongs in a process the agent can kill. dstack-util is where it goes: it already links nvml-wrapper and already calls Nvml::init during boot setup, so this adds a subcommand rather than a dependency, and an operator can run it by hand inside the CVM. One process per sample instead of a resident helper. Nvml::init runs fresh every time, so a driver that loads after the agent started is picked up on the next sample rather than being cached as "no GPU" for the agent's lifetime. stdout is the JSON document and stderr carries the log, so the subscriber is pointed at stderr for this subcommand only. The previous in-agent helper installed no subscriber at all to protect its stdout protocol, which made every NVML warning it logged a no-op.
Three problems with sampling from inside the agent: A guest with no NVIDIA card paid for the feature. Any GpuInfo call, any /metrics scrape, and any dashboard load -- including on apps with public_sysinfo off, where the result was rendered nowhere -- forked a helper that then stayed resident answering "no driver" forever. The PCI scan now runs once per process and every later request on such a guest is an atomic load returning a constant. A card present without the kernel module is reported as that, rather than as no GPU. /metrics never returned GPU data. The non-blocking path served a placeholder whenever the snapshot was older than the 5s TTL, which is every scrape at any realistic Prometheus interval: dstack_gpu_nvml_up 0 and no series at all on a healthy GPU. Callers now get the last snapshot whatever its age, with dstack_gpu_sample_age_seconds alongside, and the refresh happens behind them. The persistent line protocol had no cancellation safety. The RPC timeout sat 1s above the sampler's, and a future dropped between writing the request and reading the reply left the pipe desynchronised for good. One process per sample removes the state that could desynchronise. Sampling itself moves to `dstack-util gpu-info`; TtlCell replaces the hand-rolled snapshot cache, which is where get_allow_stale comes from.
A collector that hangs burns the sample timeout and is then killed. On the success cadence the next scrape respawned it immediately, so a guest whose driver has wedged sat in a near-continuous spawn-and-kill loop -- useless work, and the state most likely to strand a process in an uninterruptible driver call where SIGKILL only queues. Serving is unchanged: the last outcome is always returned whatever its age. Only the decision to resample is delayed. Also records what public_sysinfo now exposes. GPU UUID and PCI bus address are new identifiers on that surface and belong in the table operators read before turning the switch on.
The module justified its cache by implying sampling is expensive. It is not: one `dstack-util gpu-info` run, fork to exit, measures 94-99 ms on a single H200 in CC mode. Leaving the wrong reason in place invites the next reader to delete the cache once they measure it themselves. The reasons that survive the measurement are that `/metrics` carries CPU, memory, disk and container state that must not sit behind a wedged driver call, that three surfaces can scrape at once and would each fork their own collector, and that enumeration cost grows with card count where only one card has been measured. Recorded alongside the number. Also states what lazy refreshing means for freshness. A served snapshot is about one scrape interval old, not one TTL old, which is what the two live scrapes showed at 14.41 s and 19.96 s against a 5 s TTL. That is the design working, but it is not what the constant's name suggests, and a consumer aligning these series with a host-side exporter needs to know the offset is there.
CI failed with `refresh lock is free: TryLockError(())`. The GPU tests reach into three process-global values -- the snapshot cell, the refresh lock and the collector override -- while the harness runs them on parallel threads, so they were racing each other rather than exercising the code. Two collisions, not one. Two tests took the refresh lock with try_lock and panicked when they lost, which is the failure CI reported. A third set the collector override without holding anything, so it could point the hanging-collector test at a nonexistent path, or be pointed by it at a stub that sleeps for a minute. One guard now covers all three tests, waits instead of failing when it has to, and clears the collector override on the way out so a stub cannot leak past the test that installed it. Reproduced the original panic 4 times in 60 runs at 16 threads; the fixed tests survive 120.
The section rendered as two bare paragraphs on a page where every other block is a card, and put its values in the shape the wire uses rather than the shape a reader uses. Four things. The confidential-computing state and the sample age move into the label/value rows the rest of the page is built from, so the section stops looking unstyled. The age is rounded to one decimal, like the wattage beside it, instead of printing a raw f32. The UUID column is gone: it is forty characters that pushed every other cell into wrapping, and a reader who needs it has `/metrics` and the `GpuInfo` RPC. The PCI address drops a zero domain, because NVML writes `00000000:01:00.0` where lspci and the kernel write `01:00.0`. Both shortenings are display-only. `/metrics` still carries the full UUID and the full bus ID, because those labels are what a host-side exporter joins these series against, and a test now says so. The domain is dropped only when it is zero and only from the three-field form. A non-zero domain is what tells two cards on a multi-domain host apart, and `00:00.0` is bus zero, not a domain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Operators and the cloud control plane need to know whether a GPU attached to a CVM is actually being used: SM utilization, framebuffer usage, temperature, power, and whether the confidential-computing handshake has completed. None of that is visible from the host. With VFIO passthrough the host driver never binds the card, so host-side NVML and DCGM cannot read it, and in CC mode the device is attested to the guest only. The only vantage point is inside the guest, which is where
dstack-guest-agentalready runs. This PR gives the agent a GPU telemetry surface.Design
The first draft put a
gpuslist onSystemInfoand calledNvml::init()on every request. Review found three problems, and a survey of how DCGM exporter, go-nvml, nvml-wrapper, the Kubernetes device plugin, and the GCP/AWS in-VM agents handle this confirmed the direction:Nvmlshuts the library down onDrop, so a request-scoped handle meant init+shutdown on every/metricsscrape.ERR!state taking the whole exporter down. Here GPU collection had SysInfo's 20 s timeout andspawn_blocking; a stuck NVML call (GPU reset, driver assert, CC handshake) would stall host and guest health reporting.memory_reserved_bytes,memory_usable_bytes) were placeholders with fabricated values.ProxiedGuestApidecodes and re-encodes, so aSystemInfofield the VMM does not know is dropped silently. The VMM must ship the same proto.What this PR does instead:
GpuInfoRPC onGuestApiandProxiedGuestApi(named likeInfo/SysInfo/NetworkInfo).SystemInfois unchanged.gpu_info.rs. A failed init is cached for 60 s so driver-less images do notdlopenon every scrape.try_lock. The 5 s RPC timeout does not cancel a stuck NVML call; later callers receive the last snapshot, orerror: "sampling in progress"if there is none, instead of queueing behind it. Successful snapshots are cached for 5 s and shared by the RPC,/metrics, and the dashboard.optional. Per-deviceerrorlists the queries that failed.NotSupportedlogs at debug, because CC mode disables some counters by design; other per-field errors warn once per (device, field).uuid(stable), withindexandpci_bus_id(nvmlPciInfo.busId) alongside, matching the DCGM exporter label set so host-side tooling that speaks BDF can correlate.cc_ready(system-widenvmlDeviceGetConfComputeGpusReadyState) and per-devicecc_enabled, so "driver present but still attesting" is distinguishable from "NVML unavailable".memory_info_v2.Where the data is exposed, and who can see it
GuestApi.GpuInfoProxiedGuestApi.GpuInfoPOST /guest/GpuInfo?jsonon the VMM socket/metricson the guest's external portdstack_gpu_*seriespublic_sysinfoin app-composepublic_sysinfoin app-composeThe public surfaces reuse the existing
public_sysinfoswitch./metricsalready refused requests when it is off; the dashboard GPU section is wrapped in the same check. An app owner who keepspublic_sysinfo: falseexposes nothing new, and the host still gets telemetry through the VMM.Metrics series:
dstack_gpu_nvml_up,dstack_gpu_cc_ready, and per devicedstack_gpu_utilization_percent,dstack_gpu_memory_utilization_percent,dstack_gpu_memory_{total,used,free}_bytes,dstack_gpu_temperature_celsius,dstack_gpu_power_usage_milliwatts,dstack_gpu_query_errors, all labeledindex,uuid,pci_bus_id. A field that failed to sample emits no series rather than a 0.Tests
NonevsSome(0)and an unsetcc_ready.errorwith an empty device list, not as a silent empty success. The test passes with or without a driver present.Validation
cargo build -p dstack-guest-agent -p dstack-vmm,cargo test -p dstack-guest-agent -p guest-api,cargo fmt --check,cargo clippy -p dstack-guest-agent -p dstack-vmm -- -D warningson a TDX lab host without GPUs.next+ fix(os/mkosi): ship libnvidia-container-go.so with the container stack #1181 + chore(os): update NVIDIA driver to 595.91.07 #1177 + this branch (os_image_hasha7a7646b0111236340b576645a4c8ff29c89facf8510c1019e52383f675326a7,git_revisionclean) and the rootfs was checked offline for the NVML symbols and the 595.91.07 driver. Live single-GPU validation on that image, with a VMM built from the same tree, is the remaining step.Known limits
NotSupportedunder CC mode on B200 has not been enumerated yet; the design tolerates it but the list should be recorded once the live run happens./guest/GpuInfoto exist.References
Prior art surveyed for the design decisions above: