Repository navigation
FI_PROF_* metrics not exported (profiling module is not currently loaded) #707
Description
Ask your question
We deployed dcgm-exporter on a new node with NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, but FI_PROF_* profiling metrics are not being exported.
dcgm-exporter version: 3.3.5 (from dcgmi --version inside the container)
Driver version: 590.48.01 (from nvidia-smi)
Pod logs:
2026/07/01 16:29:57 maxprocs: Leaving GOMAXPROCS=192: CPU quota undefined
time="2026-07-01T16:29:57Z" level=info msg="Starting dcgm-exporter"
time="2026-07-01T16:29:57Z" level=info msg="DCGM successfully initialized!"
time="2026-07-01T16:29:57Z" level=info msg="Not collecting DCP metrics: This request is serviced by a module of DCGM that is not currently loaded"
time="2026-07-01T16:29:57Z" level=info msg="Falling back to metric file '/etc/dcgm-exporter/dcp-metrics-included.csv'"
time="2026-07-01T16:29:57Z" level=warning msg="Skipping line 20 ('DCGM_FI_PROF_GR_ENGINE_ACTIVE'): metric not enabled"
time="2026-07-01T16:29:57Z" level=warning msg="Skipping line 21 ('DCGM_FI_PROF_PIPE_TENSOR_ACTIVE'): metric not enabled"
time="2026-07-01T16:29:57Z" level=warning msg="Skipping line 22 ('DCGM_FI_PROF_DRAM_ACTIVE'): metric not enabled"
time="2026-07-01T16:29:57Z" level=warning msg="Skipping line 23 ('DCGM_FI_PROF_PCIE_TX_BYTES'): metric not enabled"
time="2026-07-01T16:29:57Z" level=warning msg="Skipping line 24 ('DCGM_FI_PROF_PCIE_RX_BYTES'): metric not enabled"
time="2026-07-01T16:29:57Z" level=info msg="Initializing system entities of type: GPU"
time="2026-07-01T16:29:57Z" level=info msg="Not collecting NvSwitch metrics; no fields to watch for device type: 3"
time="2026-07-01T16:29:57Z" level=info msg="Not collecting NvLink metrics; no fields to watch for device type: 6"
time="2026-07-01T16:29:57Z" level=info msg="Not collecting CPU metrics; no fields to watch for device type: 7"
time="2026-07-01T16:29:57Z" level=info msg="Not collecting CPU Core metrics; no fields to watch for device type: 8"
time="2026-07-01T16:29:57Z" level=info msg="Kubernetes metrics collection enabled!"
time="2026-07-01T16:29:57Z" level=info msg="Pipeline starting"
time="2026-07-01T16:29:57Z" level=info msg="Starting webserver"
time="2026-07-01T16:29:57Z" level=info msg="Listening on" address="[::]:9400"
time="2026-07-01T16:29:57Z" level=info msg="TLS is disabled." address="[::]:9400" http2=false
Diagnostic from inside the pod:
When trying to check DCGM modules status from within the pod, dcgmi modules -l fails to connect to nv-hostengine:
kubectl exec -it nvidia-dcgm-exporter-m7qkw -n gpu-operator -- dcgmi modules -l
Defaulted container "nvidia-dcgm-exporter" out of: nvidia-dcgm-exporter, toolkit-validation (init)
Error: unable to establish a connection to the specified host: localhost
Error: Unable to connect to host engine.
command terminated with exit code 235
I want to collect DCP metrics
- Does dcgm-exporter:3.3.5 support NVIDIA RTX PRO 6000 Blackwell?
- If yes, how to fix the Profiling module loading error?
Any help would be greatly appreciated. Thanks!
Activity
[-]FI_PROF_* metrics not exported with RTX PRO 6000 Blackwell GPUs (Profiling module fails to load)[/-][+]FI_PROF_* metrics not exported with RTX PRO 6000 Blackwell GPUs (profiling module is not currently loaded)[/+][-]FI_PROF_* metrics not exported with RTX PRO 6000 Blackwell GPUs (profiling module is not currently loaded)[/-][+]FI_PROF_* metrics not exported (profiling module is not currently loaded)[/+]rvatkar commentedon Jul 8, 2026
@tensorriccii
dcgmi --versionshows the DCGM version, not dcgm-exporter. DCGM 3.3.5 is too old for Blackwell profiling—please upgrade dcgm-exporter to 4.5.3-4.8.2-distroless or newer (releasing soon). Thedcgmi modules -lerror is expected because exporter uses embedded DCGM.