Name	Name	Last commit message	Last commit date
parent directory ..
basic	basic
minimal	minimal
README.md	README.md

SLURM Configuration Examples

This directory contains example configurations for deploying madengine workloads on SLURM HPC clusters.

📋 Convention Over Configuration

No explicit deploy field needed! The presence of a slurm field automatically indicates SLURM deployment:

{
  "slurm": {
    "partition": "amd-rccl",
    "nodes": 2,
    "gpus_per_node": 8
  }
}

⚠️ Important: The default partition is amd-rccl (for AMD RCCL clusters). If your cluster uses a different partition name (e.g., gpu, compute, batch), override it in your configuration:

{
  "slurm": {
    "partition": "your-partition-name"  // Override default
  }
}

Check your cluster's partitions:

sinfo -o "%P"  # List all available partitions

The deployment type is inferred from the configuration structure:

✅ Deployment type (k8s/slurm/local) inferred from config structure
✅ Layered defaults: base → profile → user configuration
✅ Intelligent profile selection based on node count

📁 Example Configurations

Training Configurations (`basic/`)

File	Description	Nodes	GPUs	Use Case
`01-single-node-single-gpu.json`	Single GPU training	1	1	Quick tests, small models
`02-single-node-multi-gpu.json`	Single node, 8 GPUs	1	8	Single-node distributed workload
`03-multi-node-basic.json`	2 nodes, 8 GPUs each	2	16	Multi-node distributed workload
`03-multi-node-basic-nodelist.json`	Same as 03 with `nodelist`	2	16	Pin job to specific nodes (e.g. node01,node02)
`04-multi-node-advanced.json`	4 nodes, advanced features	4	32	Production-scale training

vLLM Inference Configurations (`basic/`)

File	Description	Nodes	GPUs	Use Case
`05-vllm-single-node.json`	Single node vLLM	1	4	Single-node LLM inference
`06-vllm-multi-node.json`	Multi-node vLLM	2	8	Multi-node LLM inference with Ray

Minimal Examples (`minimal/`)

Stripped-down configurations showing only essential fields:

torchrun-single-gpu-minimal.json - Minimal single GPU config
torchrun-multi-gpu-minimal.json - Minimal 8 GPU config
torchrun-multi-node-minimal.json - Minimal 2-node config
vllm-single-node-minimal.json - Minimal vLLM single-node
vllm-multi-node-minimal.json - Minimal vLLM multi-node
slurm-multi-minimal.json - Minimal slurm_multi self-managed launcher (3 nodes)

For slurm_multi escape-hatch launcher, see Launchers Guide.

🔄 Configuration Workflow

Understanding how configurations flow through madengine:

┌──────────────────────────────────────────────────┐
│ 1. Config File (*.json)                          │
│    - Contains: slurm, distributed, env_vars      │
└──────────────────┬───────────────────────────────┘
                   │ --additional-context-file
                   ↓
┌──────────────────────────────────────────────────┐
│ 2. madengine build                           │
│    - BuildOrchestrator._save_deployment_config() │
│    - Extracts env_vars, slurm, distributed       │
└──────────────────┬───────────────────────────────┘
                   │
                   ↓
┌──────────────────────────────────────────────────┐
│ 3. build_manifest.json                           │
│    - deployment_config.env_vars (saved)          │
│    - deployment_config.slurm (saved)             │
└──────────────────┬───────────────────────────────┘
                   │ --manifest-file
                   ↓
┌──────────────────────────────────────────────────┐
│ 4. madengine run                             │
│    - RunOrchestrator._execute_*()                │
│    - Loads deployment_config from manifest       │
└──────────────────┬───────────────────────────────┘
                   │
                   ↓
┌──────────────────────────────────────────────────┐
│ 5. Docker Container Environment                  │
│    - env_vars passed to container                │
│    - SLURM job submitted with configuration      │
└──────────────────────────────────────────────────┘

Key Points:

✅ Config files are the source of truth - Don't edit build_manifest.json manually
✅ Build phase embeds configuration - Configuration is saved during build for use at runtime
✅ Run phase uses manifest - All settings come from the generated manifest
✅ Environment variables flow automatically - From config → manifest → Docker

🚀 Quick Start

1. Build-and-Run Workflow (Recommended)

When using configuration files with env_vars, use the two-phase workflow:

# SSH to SLURM login node first
ssh user@hpc-cluster.example.com

# Phase 1: Build with configuration
MODEL_DIR=models/my-model madengine build \
  --tags model_tag \
  --additional-context-file examples/slurm-configs/basic/03-multi-node-basic.json \
  --manifest-output build_manifest.json

# Phase 2: Run from manifest
MODEL_DIR=models/my-model madengine run \
  --manifest-file build_manifest.json

Why two phases?

Build phase embeds your env_vars and deployment config into the manifest
Run phase uses the pre-configured manifest
Ensures consistency across builds and deployments

2. Direct Run (For Simple Cases)

For quick tests without custom env_vars:

madengine run --tags model_tag \
  --additional-context-file examples/slurm-configs/minimal/torchrun-single-gpu-minimal.json

3. CLI Override

madengine run --tags model_tag \
  --additional-context '{
    "slurm": {
      "partition": "gpu",
      "nodes": 2,
      "gpus_per_node": 8,
      "time": "24:00:00"
    }
  }'

4. Hybrid Approach (File + CLI Override)

# Use base config, override specific fields
madengine run --tags model_tag \
  --additional-context-file examples/slurm-configs/basic/03-multi-node-basic.json \
  --additional-context '{"slurm": {"nodes": 4, "time": "48:00:00"}}'

🔄 Distributed Workload Support

The SLURM deployment automatically configures distributed execution for multi-node and multi-GPU setups (training with torchrun/deepspeed or inference with vLLM/SGLang):

How It Works

Environment Variables: SLURM sets distributed execution environment (MASTER_ADDR, MASTER_PORT, RANK, etc.)
MAD_MULTI_NODE_RUNNER: Automatically configured with the appropriate torchrun command
Docker Containers: Environment variables are passed into containers via docker_env_vars
Model Scripts: Use $MAD_MULTI_NODE_RUNNER to launch training (see below)

Model Script Pattern

Your model's run script should use the MAD_MULTI_NODE_RUNNER environment variable:

#!/bin/bash
# Example: scripts/my_model/run.sh

# MAD_MULTI_NODE_RUNNER is automatically set by madengine for distributed workloads
if [ -z "$MAD_MULTI_NODE_RUNNER" ]; then
    # Fallback for standalone execution
    N_GPUS="${MAD_RUNTIME_NGPUS:-1}"
    MAD_MULTI_NODE_RUNNER="torchrun --standalone --nproc_per_node=$N_GPUS"
fi

# Launch your Python training script with torchrun
$MAD_MULTI_NODE_RUNNER train.py --your-args

Distributed Environment Variables

The following variables are automatically available in your containers:

Variable	Description	Example
`MASTER_ADDR`	Master node address	`node001`
`MASTER_PORT`	Master communication port	`29500`
`WORLD_SIZE`	Total number of processes	`16` (2 nodes × 8 GPUs)
`RANK`	Global process rank	`0`, `1`, ...
`LOCAL_RANK`	Local GPU rank on node	`0-7`
`NNODES`	Number of nodes	`2`
`NPROC_PER_NODE`	GPUs per node	`8`
`MAD_MULTI_NODE_RUNNER`	Complete torchrun command	`torchrun --nnodes=2 ...`

Example Configurations

Single-Node Multi-GPU (Data Parallel):

{
  "slurm": {
    "nodes": 1,
    "gpus_per_node": 8
  }
}

→ MAD_MULTI_NODE_RUNNER="torchrun --standalone --nproc_per_node=8"

Multi-Node Distributed Workload:

{
  "slurm": {
    "nodes": 4,
    "gpus_per_node": 8
  }
}

→ MAD_MULTI_NODE_RUNNER="torchrun --nnodes=4 --nproc_per_node=8 --node_rank=$SLURM_PROCID --master_addr=$MASTER_ADDR --master_port=29500"

Verification

Check that distributed execution is configured correctly:

# In your SLURM output logs, you should see:
Distributed Execution Configuration:
  NNODES: 2
  GPUS_PER_NODE: 8
  TOTAL_GPUS: 16
  MASTER_ADDR: node001
  MASTER_PORT: 29500
  NODE_RANK: 0
  Launcher: torchrun (distributed)
  MAD_MULTI_NODE_RUNNER: torchrun --nnodes=2 --nproc_per_node=8 ...

🚀 vLLM Inference Configurations

vLLM is a high-throughput LLM inference engine. madengine provides pre-configured setups for both single-node and multi-node deployments.

Memory Management

vLLM configurations include critical memory management environment variables to prevent OOM (Out of Memory) errors, especially in multi-node deployments with pipeline parallelism.

Key Environment Variables

1. VLLM_KV_CACHE_SIZE

Purpose: Limits the percentage of GPU memory allocated for KV cache
Default in configs: 0.8 (80% of available GPU memory)
Why needed: Prevents vLLM from aggressively allocating all available memory, which can cause fragmentation and OOM errors
Tuning:
- Increase (e.g., 0.9) if you have large memory headroom
- Decrease (e.g., 0.6, 0.7) if experiencing OOM errors

2. PYTORCH_CUDA_ALLOC_CONF

Purpose: Configures PyTorch's CUDA/HIP memory allocator
Value: expandable_segments:True
Why needed: Reduces memory fragmentation by allowing the allocator to expand memory segments dynamically
Reference: PyTorch Memory Management

vLLM Configuration Files

Single-Node Configurations:

05-vllm-single-node.json - Full single-node config with NCCL settings
vllm-single-node-minimal.json - Minimal single-node config (in minimal/ directory)

Multi-Node Configurations:

06-vllm-multi-node.json - Full multi-node config with NCCL and Ray settings
vllm-multi-node-minimal.json - Minimal multi-node config (in minimal/ directory)

vLLM Workflow Example

# 1. Build with vLLM configuration
MODEL_DIR=models/llama2-70b madengine build \
  --tags vllm \
  --additional-context-file examples/slurm-configs/basic/06-vllm-multi-node.json \
  --manifest-output build_manifest.json

# 2. Verify memory management env_vars were embedded
grep -A 10 "env_vars" build_manifest.json
# Should show:
#   "VLLM_KV_CACHE_SIZE": "0.8"
#   "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True"

# 3. Run the inference job
MODEL_DIR=models/llama2-70b madengine run \
  --manifest-file build_manifest.json

vLLM Parallelism Strategies

vLLM automatically selects parallelism based on your configuration:

Single-Node (TP only):

{
  "slurm": {
    "nodes": 1,
    "gpus_per_node": 4
  },
  "distributed": {
    "launcher": "vllm",
    "nnodes": 1,
    "nproc_per_node": 4
  }
}

→ Tensor Parallelism (TP) = 4 across GPUs

Multi-Node (TP + PP):

{
  "slurm": {
    "nodes": 2,
    "gpus_per_node": 4
  },
  "distributed": {
    "launcher": "vllm",
    "nnodes": 2,
    "nproc_per_node": 4
  }
}

→ Tensor Parallelism (TP) = 4 within each node
→ Pipeline Parallelism (PP) = 2 across nodes
→ Requires Ray cluster for multi-node coordination

⚙️ Configuration Layers

madengine uses intelligent multi-layer configuration merging:

┌─────────────────────────────────┐
│  1. Base Defaults               │ ← slurm/defaults.json
│     (partition, time, etc.)     │
├─────────────────────────────────┤
│  2. Profile Selection           │ ← single-node.json or multi-node.json
│     (auto-selected by nodes)    │   (based on nodes count)
├─────────────────────────────────┤
│  3. User Configuration          │ ← Your config file + CLI args
│     (highest priority)          │
└─────────────────────────────────┘

Profile Auto-Selection

Single-node profile: Applied when nodes == 1
Multi-node profile: Applied when nodes > 1

📝 Configuration Reference

SLURM Section

{
  "slurm": {
    "partition": "amd-rccl",         // SLURM partition name (default: amd-rccl)
    "nodes": 2,                      // Number of nodes
    "nodelist": "node01,node02",     // Optional: restrict job to these nodes (skips node health preflight)
    "gpus_per_node": 8,             // GPUs per node
    "time": "24:00:00",             // Wall time (HH:MM:SS)
    "output_dir": "./slurm_results", // Local output directory
    "results_dir": "/shared/results", // Shared results collection
    "shared_workspace": "/shared/workspace", // Shared workspace (NFS/Lustre)
    "exclusive": true,               // Exclusive node access
    "reservation": "my-reservation", // Optional SLURM reservation name
    "qos": "high",                   // Quality of Service
    "account": "project-name",       // SLURM account
    "network_interface": "ib0",      // Network interface (ib0/eth0)
    "modules": ["rocm/5.7.0"],      // Environment modules to load
    "enable_node_check": true,       // Run node health preflight before submission (default: true)
    "auto_cleanup_nodes": false,     // Automatically clean up unhealthy nodes found during preflight (default: false)
    "allow_submit_without_clean_nodes": false, // Submit even if fewer clean nodes than requested are found (default: false)
    "verbose_node_check": false      // Print detailed node health-check output (default: false)
  }
}

nodelist: When set to a comma-separated list of node names (e.g. "node01,node02"), the job runs only on those nodes. Automatic node health preflight is skipped when nodelist is set.

Node Health Preflight: Before submitting multi-node jobs, madengine runs a health check across candidate nodes and pins the job to the healthy ones (skipped if nodelist is set).

enable_node_check (default true): Enables/disables the preflight health check.
auto_cleanup_nodes (default false): Automatically attempts to clean up unhealthy nodes found during preflight.
allow_submit_without_clean_nodes (default false): If fewer clean nodes are found than requested, submit anyway instead of failing.
verbose_node_check (default false): Print detailed health-check output during preflight.

Distributed Execution Section

{
  "distributed": {
    "launcher": "torchrun",    // Launcher type: torchrun, vllm, sglang, deepspeed, megatron, slurm_multi
    "backend": "nccl",         // Communication backend (nccl/gloo)
    "port": 29500,             // Master node port
    "nnodes": 2,               // Number of nodes (overrides slurm.nodes if set)
    "nproc_per_node": 8        // GPUs per node (overrides slurm.gpus_per_node if set)
  }
}

Supported Launchers:

torchrun: PyTorch distributed training (default)
vllm: vLLM inference engine (TP/PP parallelism)
sglang: SGLang inference engine
deepspeed: DeepSpeed training framework
megatron: Megatron-LM large model training
slurm_multi / slurm-multi: Self-managed multi-container topologies (escape hatch)
Custom: Set environment variables, model script handles launcher

Note: For vLLM and SGLang, the model script handles process spawning directly. For torchrun/deepspeed/megatron, use $MAD_MULTI_NODE_RUNNER in your model script. For slurm_multi, the model's .slurm script runs on baremetal and manages Docker containers via srun internally.

Environment Variables

{
  "env_vars": {
    "NCCL_DEBUG": "WARN",
    "NCCL_SOCKET_IFNAME": "ib0",
    "OMP_NUM_THREADS": "8",
    "MIOPEN_FIND_MODE": "1",
    "VLLM_KV_CACHE_SIZE": "0.8",
    "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True"
  }
}

Note: Environment variables set in config files are:

Saved to deployment_config.env_vars during build phase
Automatically passed to Docker containers during run phase
Available to your model scripts inside containers

🔍 Common Use Cases

Testing on Single GPU

madengine run --tags my_model \
  --additional-context-file examples/slurm-configs/minimal/torchrun-single-gpu-minimal.json

Multi-Node Training

# Build with config
MODEL_DIR=models/my-model madengine build \
  --tags training \
  --additional-context-file examples/slurm-configs/basic/03-multi-node-basic.json

# Run from manifest
MODEL_DIR=models/my-model madengine run \
  --manifest-file build_manifest.json

vLLM Single-Node Inference

# Build with vLLM config
MODEL_DIR=models/llama2-13b madengine build \
  --tags vllm \
  --additional-context-file examples/slurm-configs/basic/05-vllm-single-node.json

# Run inference
MODEL_DIR=models/llama2-13b madengine run \
  --manifest-file build_manifest.json

vLLM Multi-Node Inference

# Build with multi-node vLLM config
MODEL_DIR=models/llama2-70b madengine build \
  --tags vllm \
  --additional-context-file examples/slurm-configs/basic/06-vllm-multi-node.json

# Run multi-node inference
MODEL_DIR=models/llama2-70b madengine run \
  --manifest-file build_manifest.json

Production Deployment with Shared Storage

madengine build --tags my_model \
  --additional-context-file examples/slurm-configs/basic/04-multi-node-advanced.json

madengine run --manifest-file build_manifest.json

Custom vLLM Memory Settings

For custom memory configurations, create a new config file:

{
  "slurm": {
    "partition": "amd-rccl",
    "nodes": 2,
    "gpus_per_node": 8,
    "time": "04:00:00"
  },
  
  "distributed": {
    "launcher": "vllm",
    "nnodes": 2,
    "nproc_per_node": 8
  },
  
  "env_vars": {
    "VLLM_KV_CACHE_SIZE": "0.7",
    "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True",
    "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1",
    "HSA_FORCE_FINE_GRAIN_PCIE": "1"
  }
}

🔧 slurm_multi Launcher (Self-Managed)

The slurm_multi launcher is an escape hatch for workloads that orchestrate their own Docker containers via srun (e.g. SGLang Disaggregated proxy + prefill + decode topologies). Unlike templated launchers, the model's .slurm script runs directly on baremetal.

slurm_multi Workflow

# 1. Build with pre-built image (slurm_multi typically uses external images)
madengine build --tags sglang_disagg \
  --use-image lmsysorg/sglang:latest

# 2. Run — launcher auto-detected from model card
madengine run --manifest-file build_manifest.json

slurm_multi Inside salloc

When running inside an existing salloc allocation, slurm_multi runs synchronously with bash instead of nesting sbatch:

salloc --nodes=3 --gpus-per-node=8 --partition=gpu
madengine run --manifest-file build_manifest.json
# → Detected existing SLURM allocation: runs with bash

slurm_multi Configuration

{
  "slurm": {
    "partition": "gpu",
    "nodes": 3,
    "gpus_per_node": 8,
    "time": "04:00:00",
    "exclusive": true,
    "reservation": "my-reservation"
  },
  "distributed": {
    "launcher": "slurm_multi",
    "nnodes": 3,
    "nproc_per_node": 8
  }
}

See Launchers Guide — slurm_multi for full details.

🛠️ Advanced Features

Custom Environment Modules

Load specific software versions:

{
  "slurm": {
    "modules": [
      "rocm/5.7.0",
      "gcc/11.2.0",
      "openmpi/4.1.4"
    ]
  }
}

Shared Filesystem

Configure shared workspace and data:

{
  "slurm": {
    "shared_workspace": "/lustre/workspace",
    "results_dir": "/lustre/results"
  },
  "shared_data": "/lustre/datasets"
}

Network Configuration

For InfiniBand clusters:

{
  "slurm": {
    "network_interface": "ib0"
  },
  "env_vars": {
    "NCCL_SOCKET_IFNAME": "ib0",
    "NCCL_IB_DISABLE": "0",
    "NCCL_IB_HCA": "mlx5_0:1,mlx5_1:1"
  }
}

📊 Monitoring Jobs

After submission, monitor your SLURM job:

# Check job status
squeue -u $USER

# View job details
scontrol show job <job_id>

# View output logs (real-time)
tail -f slurm_results/madengine-*_<job_id>_*.out

# View error logs
tail -f slurm_results/madengine-*_<job_id>_*.err

# Cancel job if needed
scancel <job_id>

🐛 Troubleshooting

Job Fails Immediately

Check SLURM partition exists: sinfo
Verify GPU resources available: sinfo -o "%P %.5a %.10l %.6D %.6t %N %G"
Check SLURM account/QoS settings
Review job script: slurm_results/madengine_*.sh

Out of Memory Errors

General OOM:

Reduce batch size or model size
Use gradient accumulation
Enable CPU offloading
Check available GPU memory: rocm-smi or amd-smi

vLLM-Specific OOM (torch.OutOfMemoryError: HIP out of memory):

Symptom: Error during vLLM initialization or KV cache allocation:

torch.OutOfMemoryError: HIP out of memory. Tried to allocate 22.14 GiB. 
GPU has a total capacity of 191.98 GiB of which 145.02 GiB is free.

Root Cause: Memory fragmentation or aggressive KV cache allocation

Solutions:

Reduce KV cache size:

"env_vars": {
  "VLLM_KV_CACHE_SIZE": "0.6"  // Try 0.6 or 0.7
}

Enable expandable segments (should already be in configs):

"env_vars": {
  "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True"
}

Reduce parallelism: Use fewer GPUs or nodes for smaller models
Check GPU memory: rocm-smi or amd-smi to verify available memory
Rebuild with updated config: Don't edit build_manifest.json - update the source config file and rebuild

NCCL/Communication Errors

Verify network interface name: ifconfig or ip addr
Check InfiniBand status: ibstat (if using IB)
Test connectivity between nodes
Set correct NCCL_SOCKET_IFNAME in env_vars

vLLM Ray Connection Failures

Symptom: Failed to connect to GCS at address <node>:6379