Repository navigation
Add optional cgroup-direct process attribution for non-MIG shared GPUs #725
Description
Is this a new feature, an improvement, or a change to existing functionality?
Improvement
Please provide a clear description of the problem this feature solves
#594 added per-process metrics for shared GPUs by selecting candidate pods from kubelet PodResources and then mapping NVML PIDs to those pods through cgroups.
Our some non-MIG MPS device plugins expose logical device IDs such as:
GPU-6a16b9c2-...::5
GPU-6a16b9c2-...::19
Our shared-GPU device plugin exposes 48 logical slots per physical GPU. A workload pod usually requests 6 or 7 slots, while its processes run on one physical GPU.
These logical assignments are allocation records and may not reliably describe runtime physical GPU ownership. They can also cause the same pod-level FB_USED value to be emitted once per logical slot:
DCGM_FI_DEV_FB_USED{UUID="GPU-6a16b9c2-...",pod_uid="1843d150-...",vgpu="5",...} 10056
DCGM_FI_DEV_FB_USED{UUID="GPU-6a16b9c2-...",pod_uid="1843d150-...",vgpu="19",...} 10056
This request only covers non-MIG DCGM_FI_DEV_FB_USED.
Feature Description
As an operator using a MPS(Multi-Process Service) shared-GPU device plugin, I want an optional process attribution mode based directly on NVML PIDs and cgroups, so that pod-level GPU memory reflects actual runtime ownership instead of logical slot assignments.
Describe your ideal solution
Add an optional cgroup-direct attribution mode:
physical GPU UUID -> NVML PID -> /proc/<pid>/cgroup -> pod UID
Aggregate DCGM_FI_DEV_FB_USED by:
physical GPU UUID + pod UID
The expected pod-level output is:
DCGM_FI_DEV_FB_USED{UUID="GPU-6a16b9c2-...",pod_uid="1843d150-...",pod="workload-1",namespace="default",...} 10056
PodResources and the Kubernetes informer should not determine physical GPU ownership in this mode. The existing PodResources-based mode should remain the default. Non-FB_USED metrics should remain unchanged.
Additional context
Tested with:
dcgm-exporter: 4.5.3-4.8.2
GPU: NVIDIA RTX 6000D
Driver: 580.126.09
Environment: Kubernetes with an MPS shared-GPU device plugin
This is related to #587 and #594. I can submit a pull request if this approach is acceptable.
Activity
kvnloo commentedon Sep 27, 2026
The narrow
FB_USEDscope makes this much easier to reason about.I think the key contract for this mode should be:
physical GPU UUID + pod UID -> one aggregated FB_USED sampleregardless of how many logical MPS/device-plugin slots were assigned to that pod.
I would explicitly test the cases that can make PID/cgroup attribution misleading:
/proc/<pid>/cgrouplookup -> skip safely;I agree that PodResources should remain the default mode.
The cgroup-direct mode should be clearly opt-in because its claim is different: it reports observed runtime process ownership rather than allocation ownership.
That distinction is useful enough that I would expose the active attribution mode in logs/diagnostics as well.