Skip to content

DCGM-Exporter crashes on startup when using MIG w/ A100 #747

Closed
Closed
DCGM-Exporter crashes on startup when using MIG w/ A100#747
Labels
bugSomething isn't working

Description

@csibbitt
csibbitt
opened on Sep 22, 2026

What is the version?

4.8.4

What happened?

This is a Re-open of #443 - dcgm-exporter crashes when I enable MIG mode. Now with an even simpler reproduction.

What did you expect to happen?

I expected it to report metrics on the MIG device as described in the documentation here: https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html#multi-instance-gpu-mig-support

What is the GPU model?

A100-PCIE-40GB

What is the environment?

container on Centos 10

How did you deploy the dcgm-exporter and what is the configuration?

# podman run --name dcgm-exporter -it --device=nvidia.com/gpu=all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:4.8.4

How to reproduce the issue?

Verify a working dcgm-exporter:

[root@myhost ~]# podman run --replace --name dcgm-exporter --rm -d --device=nvidia.com/gpu=all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:4.8.4
24a8f1a60d3db64a191c4e7b7f727cedd45899ab87a60944a6d633503173b07e

[root@myhost ~]# curl -s localhost:9400/metrics | head -3

# HELP DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS Number of remapped rows for correctable errors
# TYPE DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS counter
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS{gpu="0",UUID="GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7",pci_bus_id="00000000:17:00.0",device="nvidia0",modelName="NVIDIA A100-PCIE-40GB",hostname="24a8f1a60d3d",DCGM_FI_DRIVER_VERSION="615.71.09"} 0

Enable MIG mode:

[root@myhost ~]# sudo podman rm -f dcgm-exporter
dcgm-exporter

[root@myhost ~]# nvidia-smi -i 0 -mig 1
Enabled MIG Mode for GPU 00000000:17:00.0
All done.

[root@myhost 0000:17:00.0]# nvidia-smi mig -cgi 9 -C
Successfully created GPU instance ID  1 on GPU  0 using profile MIG 3g.20gb (ID  9)
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  1 using profile MIG 3g.20gb (ID  2)

dcgm-exporter is broken:

[root@myhost devices]# podman run --name dcgm-exporter -it --device=nvidia.com/gpu=all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:4.8.4

time=2026-09-22T17:21:09.813Z level=DEBUG msg="Debug output is enabled"
time=2026-09-22T17:21:09.814Z level=INFO msg="Starting dcgm-exporter" Version=4.8.4
time=2026-09-22T17:21:09.820Z level=INFO msg="Attempting to initialize DCGM."
CacheManager Init Failed. Error: -17
time=2026-09-22T17:21:09.854Z level=ERROR msg="Failed to initialize embedded DCGM" mode=embedded enable_dcgm_log=false dcgm_log_level=NONE uid=0 euid=0 running_as_root=true has_cap_sys_admin=true error="error starting nv-hostengine: DCGM initialization error"

Disable MIG to fix dcgm-exporter:

[root@myhost devices]# nvidia-smi -i 0 -mig 0
Disabled MIG Mode for GPU 00000000:17:00.0
All done.

[root@myhost devices]# podman run --name dcgm-exporter -it --device=nvidia.com/gpu=all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:4.8.4

time=2026-09-22T17:30:09.363Z level=DEBUG msg="Debug output is enabled"
time=2026-09-22T17:30:09.363Z level=INFO msg="Starting dcgm-exporter" Version=4.8.4
time=2026-09-22T17:30:09.368Z level=INFO msg="Attempting to initialize DCGM."
time=2026-09-22T17:30:09.399Z level=INFO msg="Initialized DCGM Fields module."
time=2026-09-22T17:30:09.399Z level=INFO msg="DCGM successfully initialized!"

[root@myhost devices]# curl -s localhost:9400/metrics | head -3
# HELP DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS Number of remapped rows for correctable errors
# TYPE DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS counter
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS{gpu="0",UUID="GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7",pci_bus_id="00000000:17:00.0",device="nvidia0",modelName="NVIDIA A100-PCIE-40GB",hostname="a51576dcb9e1",DCGM_FI_DRIVER_VERSION="615.71.09"} 0

Anything else we need to know?

No response

Activity

  1. csibbitt
    added
    bugSomething isn't working
    on Sep 22, 2026

  2. nccurry commented on Sep 23, 2026

    @nccurry
    nccurry
    on Sep 23, 2026
    Collaborator

    Try running the following and let me know the results:

    # Check what the host and Podman see after creating the MIG instance.
nvidia-smi -L
nvidia-ctk cdi list

# Refresh Podman's device list after the MIG change.
sudo systemctl restart nvidia-cdi-refresh.service
nvidia-ctk cdi list

# Check GPU and MIG capability files inside the container.
podman run --rm --device=nvidia.com/gpu=all --cap-add=SYS_ADMIN \
  --entrypoint=/bin/sh nvcr.io/nvidia/k8s/dcgm-exporter:4.8.4 \
  -c 'nvidia-smi -L; ls -l /dev/nvidia-caps/'

# Retry with DCGM debug logs. Stop it with Ctrl-C if it starts.
podman run --rm --device=nvidia.com/gpu=all -p 9400:9400 \
  --cap-add=SYS_ADMIN \
  -e DCGM_EXPORTER_ENABLE_DCGM_LOG=true \
  -e DCGM_EXPORTER_DCGM_LOG_LEVEL=DEBUG \
  nvcr.io/nvidia/k8s/dcgm-exporter:4.8.4

# If it still fails, retry the last command with --security-opt=label=disable
# to check whether SELinux is blocking access.

  3. csibbitt commented on Sep 24, 2026

    @csibbitt
    csibbitt
    on Sep 24, 2026
    Author

    Thank you, I think that the systemctl restart nvidia-cdi-refresh.service was the key here. You can see how the CD device list changed, and after that I can run the exporter and it's reporting metrics as expected.

    [root@myhost ~]# nvidia-smi -L
GPU 0: NVIDIA A100-PCIE-40GB (UUID: GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7)
  MIG 3g.20gb     Device  0: (UUID: MIG-fba3af63-3842-5d44-871d-9bfc23d6a085)

[root@myhost ~]# nvidia-ctk cdi list
INFO[0000] Found 3 CDI devices                          
nvidia.com/gpu=0
nvidia.com/gpu=GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7
nvidia.com/gpu=all

[root@myhost ~]# systemctl restart nvidia-cdi-refresh.service

[root@myhost ~]# nvidia-ctk cdi list
INFO[0000] Found 5 CDI devices                          
nvidia.com/gpu=0
nvidia.com/gpu=0:0
nvidia.com/gpu=GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7
nvidia.com/gpu=MIG-fba3af63-3842-5d44-871d-9bfc23d6a085
nvidia.com/gpu=all

[root@myhost ~]# podman run --replace --name dcgm-exporter --rm -d --device=nvidia.com/gpu=all -p 9400:9400 -e "DCGM_EXPORTER_DEBUG=true" --cap-add SYS_ADMIN nvcr.io/nvidia/k8s/dcgm-exporter:4.8.4
ba11211efaf77fb84c3ed650f3a0892bc51e8318a1f70924ae19cd2ce2201ffe

[root@myhost ~]# curl -s localhost:9400/metrics | head -3
# HELP DCGM_FI_DEV_FB_FREE Framebuffer memory free (in MiB).
# TYPE DCGM_FI_DEV_FB_FREE gauge
DCGM_FI_DEV_FB_FREE{gpu="0",UUID="GPU-4b9f8464-ad90-cdde-8510-8006fc2772b7",pci_bus_id="00000000:17:00.0",device="nvidia0",modelName="NVIDIA A100-PCIE-40GB",GPU_I_PROFILE="MIG 3g.20gb",GPU_I_ID="1",GPU_CI_ID="0",hostname="ba11211efaf7",DCGM_FI_DRIVER_VERSION="615.71.09"} 19994
  4. csibbitt
    closed this as completedon Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions