Missing DRA metrics in case of dynamically created MIG devices #714

Activity

1. guptaNswati commented on Jul 24, 2026

   

   guptaNswati

   on Jul 24, 2026

   ContributorAuthor

   More actions

   Since there are a few different ways to close the Dynamic MIG mapping gap, let's first walk through how the current per-pod → DRA → DCGM metric labeling actually works.

   DCGM already knows every GPU and MIG device on the node by its NVML UUID — that's where its raw metrics come from. What it does not know on its own is which pod currently owns each device. That's the job of the PodResources + DRA translation layer inside dcgm-exporter.

   There are two inputs:

   Input 1 — Kubelet PodResources gRPC (local, on-node)
   For every pod running on this node, PodResources returns the DRA resources allocated to it as a list of (driver, pool, device) tuples https://github.com/NVIDIA/dcgm-exporter/blob/main/internal/pkg/transformation/kubernetes.go#L850:

   drInfo := DynamicResourceInfo{
       ClaimName:      dr.GetClaimName(),
       ClaimNamespace: dr.GetClaimNamespace(),
       DriverName:     draDriverName,
       PoolName:       draPoolName,
       DeviceName:     draDeviceName,
   }
   

   Input 2 — ResourceSlice informer (cluster, K8s API)
   The exporter filters and only watches the ResourceSlices published by NVIDIA DRA ("gpu.nvidia.com") driver. Using the (pool, device) tuple from Input 1, the exporter looks up the matching device in the ResourceSlice and gets four attributes — type, uuid, parentUUID, profile:

   func buildDeviceMapping(deviceType, uuid, parentUUID, profile string) (string, *DRAMigDeviceInfo) {
       switch deviceType {
       case "gpu":
           // Full GPU: the slot's UUID *is* the concrete GPU UUID.
           return uuid, nil
       case "mig":
           // MIG device: we know the parent GPU and the profile string.
           return parentUUID, &DRAMigDeviceInfo{
               MIGDeviceUUID: uuid,
               Profile:       profile,
               ParentUUID:    parentUUID,
           }
       }
   }
   

   That resolved UUID is then attached as a label to DCGM's per-device metrics streaming from NVML — so a metric like DCGM_FI_DEV_GPU_UTIL{UUID="GPU-abc…"} gets enriched with pod, namespace, container and DRAinfo.

   Some more details to already listed options:

   Option A — Parse the canonical device name in the exporter
   The (pool, device) tuple the exporter already gets from PodResources contains everything: the device string encodes parentMinor, profileID, and placementStart by construction. Reuse the driver's regex in the exporter.

   Why this is safe today, grounded in the driver:

   The name schema is not a convention or a "helpful hint" — it's constructed by MigSpecTuple.ToCanonicalName and parsed back by NewMigSpecTupleFromCanonicalName
   The driver itself relies on the round-trip in unpreparePartiallyPrepairedClaim — the reclaim-on-partial-prepare path has no other source of truth for (parentMinor, profileID, placementStart) than the device name string.
   Any future rename would already require the driver to migrate its own reclaim logic. An external consumer parsing the same schema does not add a new constraint on the driver; it depends on a constraint the driver already carries.

   Pros:
   Zero driver change. The information is already in the wire format the exporter receives from PodResources.
   No ResourceSlice attribute additions, no status.devices[].data schema, no informer/watch change on the exporter side.

   Cons:

   • The parse contract is not "documented as public API". May be expose NewMigSpecTupleFromCanonicalName as an importable api that exporter imports internally and use.
   • Still requires the NVML side-channel to resolve (parentMinor, profileID, placementStart) → MIG UUID — same as Option A. No help for future per-allocation fields (sharing / time-slicing / IMEX).
   • Full-GPU names (gpu-) don't yield a UUID by parse — for those the exporter still needs uuid from the ResourceSlice attributes (which is fine; no change from today).

   Option B — Add placementInfo (Start + Size) attributes to the ResourceSlice
   Keep the ResourceSlice as the exporter's single source of truth. Add Placement info to each Dynamic MIG device entry, alongside parentUUID and profile which exporter can use to resolve the concrete MIG UUID with a local NVML lookup similar to what it does for Static MIG.

   Pros: additive driver change; no new watch on the exporter side; no timing coupling to NodePrepareResources.

   Cons: the exporter grows an NVML side; every future per-allocation field (sharing group, time-slice partition) may also need new attributes meaning this wont work out of box for different types and sharing modes.

   Option C — Populate ResourceClaim.status.devices[].data with concrete identity
   Move concrete post-prepare identity out of the ResourceSlice (a static catalog) into ResourceClaim.status (per-allocation, post-prepare). The driver writes it in Prepare() once the MIG device actually exists.

   The exporter switches from watching ResourceSlice to watching ResourceClaim and reads the concrete identity directly for all device types.

   devices:
       - conditions:
         - lastTransitionTime: "2025-11-13T22:22:56Z"
           message: ""
           reason: GPUDeviceReady
           status: "True"
           type: Ready
         data:
           driverVersion:
             version: 1.0.0
           uuid:
             string: gpu-18db0e85-99e9-c746-8531-ffeb86328b39
         device: gpu-0
   

   Example implementation is here kubernetes-sigs/dra-example-driver#128

   Pros: concrete identity lives where it belongs (post-prepare, per-allocation); direct correlation, no NVML side-channel; naturally handles MIG destroy+recreate on the same claim slot; future-proof for sharing / time-slicing / IMEX per-allocation state.

   Cons: two-sided contract (driver publishes, exporter consumes) so both sides must ship compatible versions; exporter changes the object it watches; must handle "data not yet populated" (older driver, prepare in flight).
2. guptaNswati commented on Jul 27, 2026

   

   guptaNswati

   on Jul 27, 2026

   ContributorAuthor

   More actions

   Implementing Option A as NVIDIA DRA driver also relies on it for cleanup.
3. kvnloo commented on Sep 27, 2026

   

   kvnloo

   on Sep 27, 2026

   More actions

   Option A sounds like the strongest direction to me if the NVIDIA DRA driver already needs the concrete mapping for cleanup.

   I would make one invariant explicit before implementing it:

   (driver, pool, abstract device) -> concrete live MIG UUID

   must come from an authoritative post-PrepareResources mapping.

   I would avoid reconstructing the concrete device only from:

   parentUUID + profile

   because that becomes ambiguous as soon as one parent contains multiple live MIG instances with the same profile.

   A regression that seems important here is:

   1. create two same-profile MIG devices on the same parent GPU;
   2. allocate each to a different pod through DRA;
   3. verify DCGM metrics are attributed to the correct pod for both devices;
   4. tear one claim down and recreate it;
   5. verify the old concrete UUID mapping cannot leak into the new allocation.

   That tests the exact case where heuristic profile matching would look correct until attribution silently swaps between identical slices.