vllm_disagg: GLM-5.2 (FP8 + MXFP4) on MI355X/gfx950 + AI NIC, 1P/1D EP8 [stacked on #176] - #205
Open
raviguptaamd wants to merge 41 commits into
Open
vllm_disagg: GLM-5.2 (FP8 + MXFP4) on MI355X/gfx950 + AI NIC, 1P/1D EP8 [stacked on #176]#205raviguptaamd wants to merge 41 commits into
raviguptaamd wants to merge 41 commits into
Conversation
…ement Adds GLM-5.1-FP8 (GlmMoeDsaForCausalLM = MLA + DeepSeek Sparse Attention) to the MoRI-EP WideEP disaggregated serving path, stacked on the ROCm#171 unified launcher. Fully isolated from DeepSeek-V3/R1: GLM gets its own image + a MODEL_NAME-gated runtime path, so existing models are byte-identical to develop. Defects fixed (validated 1P/1D EP8 + 2P/2D EP16, NIAH 2k-35k = 10/10, no crash): - Long-context accuracy collapse: vLLM #47766 cache-key fix keeps the persistent sparse-MLA kernel ON (keys metadata on per-request context+query len). - 8k disagg prefill crash: DSA adds a 2nd (indexer) KV cache per layer that the single-geometry MoRIIO connector never transferred; paired + shipped prefill-> decode. Plus DSA invalid-token kernel fix (#45324) and shik-latest DP-notify. Changes: - docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile: NEW per-model image (raviguptaamd/vllm glm5.1-dsa-wideEP_on_shik_latest + aiter e03fa6040 + mori 42e895472b08 + router). The base vllm_disagg_inference Dockerfile (DSV3/R1) is left untouched. Future models add their own Dockerfile the same way. - models.json: card pyt_vllm_disagg_mori_glm-5.1-fp8 (GLM_SKIP_PATCHERS=1: image carries the DSA fixes in-source). - models.yaml: GLM-5.1-FP8 recipe (block=1, AITER MLA on, eager, mori backends). DeepSeek-V3 dp: caps (--max-num-seqs 64 --max-model-len 32768) to bound the newer base's decode logits workspace (isolated to the DSV3 entry). - connectors/moriio.sh: MODEL_NAME-gated GLM DSA runtime patchers (pure no-op for other models); GLM_SKIP_PATCHERS switch for baked-fix images. - 9 idempotent, anchor-based, self-skipping GLM DSA patcher scripts. KNOWN OPEN DEFECT (future work): 4P/4D EP32 emits corrupted tokens at all context lengths (suspect moriep all-to-all combine at scale); use 1P/1D and 2P/2D. Co-Authored-By: Claude <noreply@anthropic.com>
benchmark_niah.py mis-scored thinking models: it never disabled thinking and read only content + reasoning_content. GLM-5.1 emits chain-of-thought into the `reasoning` field and leaves `content` empty until the final answer, so with a small max_tokens the answer never lands in content -> a false 0/10 even when generation is correct. - Add chat_template_kwargs.enable_thinking=false so the answer goes to content. - Also score the `reasoning` field as a fallback. Verified against GLM-5.1-FP8: correct 9-10/10 retrieval across 2k-35k on all tested topologies (EP8/EP16/EP32) after the fix. Co-Authored-By: Claude <noreply@anthropic.com>
…accuracy Needle layout is seeded, so a single run is deterministic (bit-exact on the same stack) but can't tell a real accuracy dip from single-needle variance. Add NIAH_SEEDS (default 0,1,2) to run each context length across multiple needle layouts; the summary now reports mean/min/max across seeds. Backward compatible: NIAH_SEEDS=0 reproduces the prior single-seed behavior. Co-Authored-By: Claude <noreply@anthropic.com>
…robe) On a freshly-booted node the first request of each context shape pays the full JIT/kernel-autotune compile (minutes). The NIAH harness scored the FIRST request, so cold compile landed on a scored/gated request -> false 0/10 or timeout, failing the accuracy gate and skipping the perf sweep. Root-caused by reproducing on a cold boot (0 results) vs a warm server (all pass) on the same image. Fixes: - benchmark_niah.py: add a warmup pass (NIAH_WARMUP=1 default) — one throwaway request per context length before scoring, with a generous timeout, failures tolerated. Scored requests are then always warm. - benchmark_niah.py: distinguish TIMEOUT/ERROR from a wrong answer. Timeouts return a sentinel (excluded from mean, never counted as 0/10); summary flags NO-RESULT with guidance instead of silently reporting 0. - benchmark_niah.sh: replace the blind `sleep 10` with a /v1/models readiness poll (up to 5 min), and forward NIAH_WARMUP. Verified: patched harness on the warm server passes 10/10; cold-boot repro no longer produces false 0/10 because compile happens in the warmup pass. Co-Authored-By: Claude <noreply@anthropic.com>
…launcher fixes Validated on MI300X, 8 nodes, image rocmshared/pytorch-private:glm5.1-vllm027-b8 (base ci_base-dedbf6be8b + vLLM raviguptaamd/vllm@glm5.1-dsa-wideEP_on_vllm-v0.27 + aiter e03fa6040 + MoRI 42e895472b08 + router ROCm#181). PERF FIX (models.yaml decode.dp) -- the headline change: --max-num-batched-tokens 2048 on the DECODE role only. max_num_batched_tokens is a chunked-prefill SCHEDULER knob, but it also sizes the MoRI EP dispatch buffer (fused_moe/layer.py -> all2all_utils.py -> all2all.py max_num_inp_token_per_rank). At the 8192 default a decode instance ran an 8192-token-wide all2all every step, per layer, x78 layers, while decoding a handful of tokens: a fixed ~302ms/step floor, ~320x this model's HBM-bandwidth bound. Prefill keeps 8192 (it genuinely dispatches wide batches). 1024/64 con=8, warm: TPOT TTFT out tok/s 1P/1D 302 -> 88.0 ms 2431 -> 906 ms 24.9 -> 78.8 2P/2D 302 -> 94.1 ms 1633 ms 66.7 Published reference: 1P/1D ~89ms, 2P/2D ~91ms -> matched within 3%. Accuracy unaffected: NIAH 2k-200k clean on both topologies, no length collapse, memfault=0, latencies equal-or-better at every length. 200K validated (5.7x beyond the previously published 35K ceiling). Dockerfile: base -> ci_base-dedbf6be8b (matches the fork's upstream base), VLLM_REF -> the v0.27 branch, and WITH_MORI_BUILD/WITH_AITER_BUILD now default to 1 so a plain `docker build` reproduces the validated stack. Previously they defaulted to 0, which silently used the base's bundled aiter 0.1.19 -- that GPU-faults on the GLM DSA decode kernel. The pinned aiter e03fa6040 / MoRI 42e895472b08 must not be bumped without re-running long-context NIAH. connectors/moriio.sh: per-role env split (PREFILL_*/DECODE_* -> VLLM_MORI_*), mirroring the existing PREFILL/DECODE_MORI_BACKEND pattern -- models.yaml env: applies to BOTH roles, but prefill and decode need opposite values here. Also injects use_inductor_graph_partition (pairs with the vLLM splitting_ops fix). connectors/moriio.env: RDMA fabric -- MORI_IB_GID_INDEX=3 (RoCEv2 IPv4), MORI_RDMA_DEVICES/NCCL_IB_HCA restricted to the 8 GPU-local NICs (leaving the mgmt NICs in makes QPs form over a non-routable fabric -> ibverbs.cpp:189 timeouts), NCCL/GLOO control sockets on eth0. run_xPyD_models.slurm: libionic bind-mount requires a regular file after symlink resolve (a dangling symlink gave "OCI runtime create ... not a directory", container exit 125); prefer FABRIC_SUBNET over `hostname -I` first IP (nodes list a 10.224 overlay first, which made the socket_barrier advertise an unreachable NIC -> "Waiting for nodes" hang); GLM_KERNEL_PATCH/GLM_BACKEND_PATCH bind-mount hooks to test .py fixes without a rebuild; forward the new per-role env keys. vllm_disagg.sh: same FABRIC_SUBNET IP-selection fix for host_ip. benchmark_xPyD.sh: per-shape warmup at the REAL isl/osl before each shape's cells. The global warmup is isl=osl=32/con=1, which never exercises a 1024/8192/28672 prefill path or the decode cudagraph batch sizes, so the first measured cell absorbed residual JIT (observed 302ms vs ~88ms steady-state). Warmup output goes to a separate _SHAPEWARMUP.log so it cannot pollute the CSV. models.yaml (GLM-5.1-FP8): decode.dp perf fix above; recipe = prefill eager + mori_high_throughput, decode PIECEWISE cudagraph + mori_low_latency; VLLM_USE_LAYERNAME=0; VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=64; NCCL heartbeat/timeout knobs for long-context collectives. Full operational playbook (including the dead ends) in skills_vllm_disagg.md. Co-Authored-By: Claude <noreply@anthropic.com>
GLM-5.1 is served over MoRI-EP (all2all) + MoRI-IO (KV transfer); the rixl connector's UCX/RIXL/rocSHMEM/DeepEP stack is never selected by this recipe, so building it only lengthens the build and ships unused transports. 0 is also how the validated image was actually built: glm5.1-vllm027-b8 used --build-arg WITH_NIXL=0, so the previous default of 1 meant a plain `docker build` did NOT reproduce the tested artifact. With this change, and with WITH_MORI_BUILD/WITH_AITER_BUILD already defaulting to 1, a no-flag build now matches the validated stack exactly. Set --build-arg WITH_NIXL=1 if you need the rixl connector from this same Dockerfile. Co-Authored-By: Claude <noreply@anthropic.com>
…ybook
niah_200k.py: needle-in-a-haystack sweep that validated GLM-5.1-FP8 to 200,049 tokens
on both 1P/1D (EP8) and 2P/2D (EP16). Reports found/10, latency, and the server-reported
prompt_tokens per length, and writes JSON. Model id is overridable via NIAH_MODEL so it
is not GLM-specific. The existing benchmark_niah.* stop well short of this range; this
covers the 64k-200k band.
skills_vllm_disagg.md: operational playbook for vLLM PD-disaggregated WideEP on MI300X
(MoRI-EP + MoRI-IO), written from this enablement. Documents, with measurements:
- benchmarking method: ALWAYS discard the first post-boot run (cold Triton JIT made
TTFT read 13.4s vs 906ms warm; with prefill eager the JIT cost lands in TTFT, not
TPOT), and sanity-check against the HBM-bandwidth bound before blaming a kernel
- the max_num_batched_tokens trap: a chunked-prefill SCHEDULER knob also sizes the
MoRI EP dispatch buffer, so decode ran an 8192-token-wide all2all every step
(302ms -> 88ms TPOT once sized for the real batch)
- the DSA sentinel landmine: the invalid sparse-index sentinel must be 0, not -1,
because aiter's mla_decode_fwd dereferences it (only bites at disagg long context)
- three documented DEAD ENDS so they are not retried, including why mori's
max_total_recv_tokens cannot decouple recv from send capacity (the clamp is a min())
- cache/boot behaviour (three caches with different rules, the aiter baton lock,
measured boot times), readiness signals for multi-node topologies, per-role env
plumbing, and RDMA fabric verification (a node can be SLURM-'alloc' with a dead
fabric - verify with ping/ib_write_bw before blaming code)
Co-Authored-By: Claude <noreply@anthropic.com>
…ments
Review cleanup. The VLLM_MORI_MAX_TOKENS_PER_RANK / VLLM_MORI_MAX_TOTAL_RECV_TOKENS
per-role plumbing was written while chasing the decode-TPOT bug and is NOT what fixed it
(the fix is `--max-num-batched-tokens 2048` on decode.dp). Worse, the comments asserted
that max_total_recv_tokens keeps recv capacity large enough for vLLM's profiling dummy
run -- which is false and was disproved by measurement: mori's
MaxNumTokensToRecvPerRank() = min(ceil(maxTotalRecvTokens / worldSize),
maxNumInpTokenPerRank)
is a min(), so maxTotalRecvTokens can only LOWER recv capacity, never raise it above the
send width. Anyone following those comments and setting the knobs would hit
"Total recv token overflow" at boot (observed at 512, 2048, and with recv=65536).
Removed: the per-role export block in moriio.sh, the stale models.yaml comment block, and
the six dead keys from _RECIPE_ENV_KEYS. Replaced with a short NOTE in moriio.sh pointing
at the real fix and at skills_vllm_disagg.md for the measurements and dead ends.
No functional change to the validated configuration: the knobs defaulted to 0/unset, so
the b8 runs never exercised them.
Co-Authored-By: Claude <noreply@anthropic.com>
scripts/vllm_dissag/.nfs0000000016f44d2b00008188 was committed by accident. It is a stale byte-copy of run_xPyD_models.slurm (same #!/bin/bash, #SBATCH --job-name=vllm-pd, #SBATCH -N 2 header) left behind by NFS when the real file was rewritten while an open handle still referenced it. Nothing reads it and it will drift from the real launcher on every edit.
… a silent wrong-arch bug
Targets MI355X and the Pensando Ionic AI NIC:
GFX_COMPILATION_ARCH gfx942 -> gfx950
NIC_COMPILATION_ARCH cx7 -> ionic
MORI_GPU_ARCHS gfx942 -> gfx950
MORI_DEVICE_NIC (new) = ionic
MORI_DEVICE_NIC is pinned rather than left to auto-detect. Detection does resolve
correctly here (8x rocep*s0 -> driver readlink -> ionic, with
libionic.so.1.1.54.0-184 present), but pinning keeps the device-side IBGDA JIT
path deterministic across nodes.
Renames two build ARGs, which is the load-bearing part of this commit:
ARG PYTORCH_ROCM_ARCH -> ARG BUILD_ROCM_ARCH
ARG MAX_JOBS -> ARG BUILD_MAX_JOBS
(both assigned through to the original ENV names)
The base image (rocm/vllm-dev:ci_base-*) exports PYTORCH_ROCM_ARCH and MAX_JOBS as
ENV, and an inherited ENV overrides a same-named ARG in the child Dockerfile. So
`ARG PYTORCH_ROCM_ARCH="gfx950"` was being silently ignored and the build used the
base's `gfx90a;gfx942;gfx950` instead.
This surfaced only by luck: the base also sets MAX_JOBS to the empty string, so
setup.py::compute_num_jobs crashed on int(''). Had it been non-empty the build
would have "succeeded" while compiling for the wrong arch list, and every
downstream perf number would have been measured on an unintended binary with
nothing in the logs to say so. Confirmed fixed by the rebuild log:
"84 warnings generated when compiling for gfx950".
Also replaces the post-install verification heredoc with an equivalent
`python3 -c "$(printf ...)"` form, and reads the vLLM version via
importlib.metadata instead of `import vllm` (importing pulls torch -> amdsmi ->
libamd_smi.so, which is not loadable in the no-GPU build sandbox).
…, disable 2 harmful patchers
Three independent fixes to the MoRIIO connector wiring.
1. qp_per_transfer / num_workers / post_batch_size now go through
kv_connector_extra_config instead of VLLM_MORIIO_* env vars, which the
connector no longer reads. moriio_common.py:227-229 holds a rename map and
_warn_deprecated_env_vars() only warns; moriio_engine.py reads no env vars at
all; vllm/envs.py has no MORIIO entries. The values are consumed solely at
moriio_common.py:342-344 via extra_config.get(), defaulting to 1/1/-1.
So the env vars set in moriio.env were silently inert, and the live log read:
Using MoRIIO backend: RDMA (qp_per_transfer=1, post_batch_size=-1, num_workers=1)
i.e. one queue pair and one worker thread for the whole prefill->decode KV
handoff on a node with 8 RDMA rails. Grep that line in the decode log to
confirm the values now change.
Also passes host_ip explicitly. The connector otherwise derives it from
get_ip(), which on this cluster returns the non-routable public address, so
decode cannot notify prefill of block allocation and the two deadlock.
2. The DSA patcher gate goes from an exact "GLM-5.1-FP8" match to the whole GLM
DSA family (GLM-5.1-FP8 | GLM-5.2-FP8 | GLM-5.2-MXFP4). These are the same
architecture -- GlmMoeDsaForCausalLM, 78 layers, 256+1 experts, index_topk=2048,
verified by diffing config.json -- so they need identical patchers. Under the
old exact match GLM-5.2 silently skipped all of them.
3. Two patchers become opt-in (default OFF). Both are actively harmful on this
image; each retains a full rationale in-line, summarised here:
apply_glm_dsa_persistent_kernel_gate_fix.py (GLM_PERSIST_GATE=1 to restore)
Sets work_meta_data=None on chunked-prefill continuations, routing to a
non-persistent kernel that does not exist for this cell: gqa_ratio=64 +
fp8/fp8 hits AITER_CHECK(false) at asm_mla.cu:945 -> C++ abort, no Python
unwind, server dies. Explains the reproducible 2000-ok / 8000-dies boundary
(8192 = prefill max_num_batched_tokens). The image already carries aiter#3921
and vllm#47766, which superseded #47567 and whose diff is the literal inverse
of this patcher.
apply_glm_dsa_kernel_fix.py (GLM_DSA_SENTINEL_FIX=1 to restore)
Flips the DSA invalid-token sentinel 0 -> -1 per vllm#45324. This image ships
0 deliberately: aiter's decode-only mla_decode_fwd sparse kernel dereferences
paged_kv_indices, so -1 becomes kv_cache + (-1)*stride -> memory access fault,
and it only bites under disagg, which is what we run. Applying it killed
decode DP0 with hipErrorIllegalAddress on a 5-token warmup curl. With
index_topk=2048 a short prompt is the worst case, not the safest.
…r MI355X
Adds the two GLM-5.2 recipes and corrects two stale values inherited from the
GLM-5.1/MI300X recipe they were derived from.
New: GLM-5.2-FP8 (zai-org/GLM-5.2-FP8) and GLM-5.2-MXFP4 (amd/GLM-5.2-MXFP4),
1P/1D EP8 over MoRI-IO. Measured on 2x8 MI355X at ISL/OSL 28672/1024:
TPOT 33.7 / 36.4 / 40.2 ms at concurrency 16 / 32 / 64, against a 50 ms target.
GLM-5.2-MXFP4 is CONFIG-ONLY AND HAS NEVER BEEN BOOTED. It is a copy of the FP8
recipe with the two values that provably do not transfer re-derived. Treat its
first run as bring-up, not benchmark.
Two values carry the recipe; both are noted in full in-file.
DECODE_CUDAGRAPH_MODE: FULL_AND_PIECEWISE (was PIECEWISE)
The single change that fixed TPOT: 104 -> 34-40 ms. PIECEWISE splits the decode
graph on the 3 DSA ops x 78 layers, ~234 launch boundaries per step. The cost is
batch-INDEPENDENT, so it presents as a latency floor that does not move with
concurrency, batch size or fabric -- which is why it survived so much tuning
before being found. Co-requisite: use_inductor_graph_partition must stay ON and
no bare --enforce-eager, else boot and warmup both pass and the first real MLA
decode dies.
prefill.dp: --gpu-memory-utilization 0.72
Prefill only. The MLA chunked-prefill workspace is sized from a hardcoded
64k-token clamp (determine_chunked_prefill_workspace_size,
mla_attention.py:1935), NOT from max_num_batched_tokens, so it allocates
65536*64*(192+256)*2 = 3.50 GiB on top of the KV pool regardless of scheduler
batch size -- and lazily, on the first long prefill, i.e. after boot and warmup
have both reported healthy. At 0.80 this OOM'd at mla_attention.py:739.
Corrections to the shared GLM section:
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB: 64 -> 2048
64MB was an MI300X crash workaround, and the crash cannot occur here: the
workaround kernel is behind a hard arch gate (rocm_aiter_mla_sparse.py:591
dispatches fp8_mqa_logits_gfx942 only if _ON_GFX942). Verified live in-image:
_ON_GFX942=False, arch=gfx950. The module self-describes as a "Temporary gfx942
fallback" for that part's 64 KiB LDS budget. Keeping 64MB cost 50 indexer
sub-chunks at ISL 28672 where 2048MB needs 2, each a full kernel chain over 78
layers. 2048 not 4096 deliberately: nearly all the reduction at half the peak
memory (2.00 GiB, against ~31 GiB/GPU idle).
KV_BLOCK_SIZE comment corrected (value unchanged at 1)
It claimed the DSA sparse indexer "REQUIRES" block-size 1. That inverts the real
constraint: rocm_aiter_mla_sparse.py:283 returns [1, 64] and indexer.py:137
returns `[1, 64] if current_platform.is_rocm() else [64]` -- ROCm is the
PERMISSIVE branch and it is non-ROCm that is locked to 64. Measured 1 vs 64:
<2% either way, so 1 stays as the validated default, but the comment would have
stopped anyone from trying 64.
The same section's "decode PIECEWISE" prose is updated to match the new value.
…/ionic env file
Everything needed to run this recipe on a second cluster without editing the
launcher. Previously several site facts were baked in as defaults from the MI300X
cluster it was written on.
FABRIC_SUBNET is the important one. It defaults to 10.158. (the MI300X fabric);
AAC is 10.2.80.. When it does not match, the awk picker falls through to the first
address on the line -- a 10.224.x overlay -- MASTER_ADDR is advertised as something
the peer cannot reach, and the run hangs at "Waiting for nodes" forever with no
error. So a fresh clone of this recipe would have hung on AAC.
Unlike every other key in the connector .env files, FABRIC_SUBNET is consumed by
the launcher on the HOST rather than inside the container, and the .env loader only
built `-e KEY=VAL` docker args. Setting it in the .env file would therefore have
been silently inert. The loader now has an explicit one-key host-export allow-list.
Deliberately an allow-list and not a blanket export: the other keys are HSA/RDMA
runtime settings that should not leak into srun and other host-side tooling.
Submit-time `export FABRIC_SUBNET=` still wins over the file, as before.
Also in the launcher:
- CONNECTOR_ENV_FILE is overridable, so a site can supply its own platform env
(e.g. connectors/moriio.env.aac) without touching the tracked default.
- CONTAINER_CLI auto-selects docker or podman by probing for a usable docker
daemon. Compute nodes here deny the daemon socket and only have podman. The
two differ in ways that matter: podman gets --group-add keep-groups (video
is not a valid group there) and no --shm-size (it does not accept it in this
configuration).
- MODEL_DIR defaults to /shared/data/amd_int/models when that exists.
- Optional site mounts (/shared/data, /shared/apps, /media/NVME, ...) are mounted
only when present, instead of failing on absence.
- The JIT cache host path is chosen from a candidate list keyed by image ID, and
falls back with a loud warning rather than silently paying a cold compile every
run.
- GLM-5.2-FP8 and GLM-5.2-MXFP4 added to the three model allow-lists.
- New niah / niah_perf benchmark cases.
connectors/moriio.env.aac (new) is the AAC MI355X + Pensando Ionic platform env:
8 ionic rails (rocep{9,25,105,121,137,153,233,249}s0) at MORI_IB_GID_INDEX=1,
control/bootstrap on the mgmt NIC via MORI_SOCKET_IFNAME=enp193s0f1np1, and
FABRIC_SUBNET=10.2.80..
The interface choice is not the obvious one and the file records why: the
public/default-route NIC answers ICMP but has TCP firewalled node-to-node, so
"the one that pings" picks the wrong link. Test the actual port -- measured
gpu-44 -> gpu-45, /dev/tcp/<public>/22 FAILS while /dev/tcp/10.2.80.16/22 succeeds.
MORI_SOCKET_IFNAME left undefined fell through to a hardcoded eth0 that does not
exist here, ShmemGetUniqueId failed, and the null shmem context then segfaulted
inside EpDispatchCombineHandle -> ShmemNumQpPerPe().
…er needle placement benchmark_niah.py printed a score table and always exited 0, so an accuracy regression could not fail a run -- it had to be spotted by eye. It now exits non-zero: 3 when a context length produced no usable result, 4 when mean retrieval is below NIAH_MIN_SCORE (default 8.0/10), 0 otherwise, with an explicit "NIAH VERDICT: PASS/FAIL" line and a per-row marker on the offending lengths. Needle placement was fully deterministic: every needle sat at exactly (i+1)*step, the same offsets for every seed, so the documented NIAH_SEEDS sweep was re-testing one identical layout and its mean/min/max spread across seeds could not show what it claimed to. Needles are now jittered within their slot (seed 0 keeps the exact old placement, so existing baselines stay comparable). Adds benchmark_niah_perf.sh: the same NIAH retrieval scoring driven at concurrency so accuracy is checked under load rather than only single-stream. Known gap, not addressed here: needles are placed at (i+1)*step for i in 0..len(ANIMALS)-1, so nothing is ever placed in the final step-width of the context. The last chunk -- often the interesting one for a long-context regression -- is never probed.
… backend and arch tags
Three reporting bugs. None affected the served workload, all affected the numbers
that get filed.
1. The first (lowest-concurrency) cell of every run was silently discarded. The
parse loop guarded its config lookup with `if i > 1`, which reads like a bounds
check but is not one: enumerate starts at 1, so i-1 is 0, a valid index --
sections[0] is the preamble that holds the first cell's [RUNNING] line. A 21-cell
sweep produced 20 rows. Regression-checked with a synthetic 3-cell log.
2. Rows with no parseable config or throughput were dropped silently, which made
this parser bug look like a benchmark-loop bug. They now print a warning naming
the result index.
3. Two hardcoded tags:
backend fell back to 'nixl'. The real selector is CONNECTOR={rixl|moriio},
resolved and exported by run_xPyD_models.slurm:188-237; RUN_MORI/RUN_DEEPEP are
the legacy flags mapped onto it there. So every run driven the current way --
including all the MoRIIO ones -- was filed as nixl. Now reads CONNECTOR,
defaulting to moriio, with the legacy flags still taking precedence.
gpu_architecture was hardcoded 'gfx942', so MI355X runs were filed as MI300X.
Now from GPU_ARCHITECTURE, defaulting to gfx950 (this recipe's target). It is
not introspected because this parser usually runs on the head node, which has
no GPU.
Adds GLM52_MI355X.md (invocation, measured numbers, the two load-bearing config values, the cluster-specific traps) and extends skills_vllm_disagg.md with the perf-tuning findings from this bring-up. Records the results honestly, including what is not solved: TPOT 33.7 / 36.4 / 40.2 ms at concurrency 16 / 32 / 64 -- target 50 ms, met. Prefill is 6,135-8,359 tok/s/rank against a 34,000 target, ~4-5x short, and is now the binding constraint. Profiled at 8192 tokens: MLA sparse attention ~82%, all GEMM 17.8%, of which MoE 2.9%. At 80K ISL the fitted single-request TTFT is ~13.7 s on one rank before any concurrency, against a <7 s SLO. TTFT at concurrency is drain time (458,752 tok / 39,340 tok/s = 11.66 s, an identity that reproduces the measurement), so more prefill nodes fix concurrent TTFT but do nothing for the single-request number. That needs tuned MLA sparse-attention kernels for gfx950 or intra-request sharding. Also records the levers closed BY MEASUREMENT so they are not re-litigated: MoE and AITER tuning (2.9% of prefill), indexer sub-chunking (<=2.5%), max_num_batched_tokens (<=4%), RDMA QP/worker count (-1.9%, inside +-0.4% noise), FP8BMM (~0.7%), block_size 1->64 (<2%). And why PCP is not the near-term answer: config/parallel.py:528 is DP-incompatible, rocm_aiter_mla_sparse.py has no PCP plumbing at all, and platforms/rocm.py:894 force-sets cudagraph_mode=PIECEWISE, which would silently undo the TPOT fix. Two constraints called out because both fail in a way that looks like success: use_inductor_graph_partition must stay ON and no bare --enforce-eager (boot and warmup pass, the first real MLA decode dies), and the MXFP4 recipe has never been booted.
raviguptaamd
requested review from
Rohan138,
amathews-amd,
coketaste,
gargrahul and
ppalaniappan-amd
as code owners
August 17, 2026 00:22
…oodput-gated) benchmark_xPyD.sh is a tuning sweep and cannot answer "do we meet the SLO". This adds a harness that can, and exits non-zero when we do not. Four differences from the sweep, each of which was producing a wrong answer: 1. --request-rate inf is a saturation test. It fires everything at once, so TTFT is just queue drain time -- at con=64/isl=28672 the measured TTFT is exactly 458,752 tok / 39,340 tok/s = 11.66 s, an identity rather than a property of the model. The customer's "<7 s avg TTFT" is about a *served* request, so this runs at a finite Poisson rate derived from Little's Law: rate = concurrency / SLO latency. That asks "can it serve the offered load AT the SLO", which is the actual question. 2. The sweep reports mean/max; the customer asked for p50/p95/p99. Now passes --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99. 3. Pass/fail was a human reading a log. The SLO is now handed to vLLM directly via --goodput ttft:7000 tpot:50, so the fraction of traffic actually served acceptably is a number the server computes. A cell can have a passing mean TTFT while a third of its requests miss; goodput catches that and the mean does not. 4. --random-prefix-len 0 makes every request unique. The customer's use case is agentic and they flagged interest in long prefix caching -- an agent loop re-sends a large static context every turn. vLLM's RandomDataset builds the prefix once and shares it across requests, so PREFIX_FRAC (default 0.5) models that. 0.5 is a declared assumption, not a measurement: the customer gave no reuse ratio, so sweep it. Scenarios are the customer's own: 80K/1K (256K context) and 200K/1K (1M context), concurrency to 256/DP. The 200K concurrency list stops at 64 on purpose. GLM-5.2 MLA KV is 43.88 KiB/token (kv_lora_rank 512 + qk_rope 64, FP8, 78 layers), so 200K = 8.37 GiB/request against a ~930 GiB single-node pool -- concurrency 256 would need ~2,142 GiB and cannot fit on one decode node. Defaulting to a list that fits means a failure is a real failure and not a predictable OOM. The arithmetic is in the header so the ceiling can be recomputed rather than rediscovered. slo_report.py divides throughput by --dp-ranks because the customer's targets are per-rank and vLLM reports aggregate; not doing so overstates by the DP degree. It labels prefill tok/s as an end-to-end lower bound (total_input_tokens/duration, which includes decode time for the same requests) rather than passing it off as an isolated prefill-engine number.
…backend
The customer sheet gives an AVERAGE ISL and a context window. gen_workload.py turns
that into a lognormal whose mean is the average and whose p99 lands on the window --
but the realised mean of a single few-hundred-request draw from a right-skewed
distribution is a property of the seed, not of the workload:
CV = sqrt(exp(sigma^2)-1), SE(mean)/mean = CV/sqrt(n)
256K row: sigma 0.5833 -> CV 0.637, n=256 -> 4.0%
1M row: sigma 0.8778 -> CV 1.078, n=128 -> 9.5%
Measured over 12 seeds the realised means actually spanned 14.2% and 35.4%. Quoting
one run's mean would be quoting that seed.
RESAMPLE_PER_ITER=1 makes each iteration draw a FRESH sample (seed = SEED_BASE +
iter-1) so iterations POOL and the error falls as 1/sqrt(iters) -- to 1.3% and 3.0%
at ten. The previous behaviour (one sample replayed) is kept as the default because
it answers a different, also-valid question: server variance on a fixed workload.
Averaging one sample with itself cannot shrink its sampling error.
benchmark_avg_{80K,200K}_ten.sh are thin wrappers that set exactly that. All
measurement logic (Little's-Law rate, goodput thresholds, timeouts, warmup) stays in
benchmark_customer_slo.sh with one implementation; two copies would drift and the
drift would surface as an unexplained delta between the two rows.
pool_workload.py pools RAW LENGTHS and re-computes. Percentiles do not average: the
mean of ten p99s is not the p99 of the pool, and for a heavy tail it is materially
lower. It refuses to pool samples with different targets, because a pooled 80K+200K
sample describes no workload the customer asked about. Pooled results: 80K row
+0.6%, 200K row -1.4%, both p99 landing exactly on the stated window.
Stratified inverse-CDF placement was considered and rejected. It pins the mean to
1-2% with no pooling, but caps the sample at the (n-0.5)/n quantile, so p99 would
reach ~225K instead of 262,144. The window is the number the customer actually
stated; we do not trade it away to tidy a number they did not.
Also unhardcodes the driver so this runs against something other than vLLM:
BENCH_BACKEND (default vllm), BENCH_EXTRA_ARGS, and IGNORE_EOS. --ignore-eos is a
vLLM extension and is what makes OSL exact; dropping it means the model stops when
it wants to and the measured TPOT averages over a length we did not choose, so the
50 ms/token verdict becomes approximate. The header says so, because the risk is
that someone quotes it as the sheet's number anyway.
Two container defects fixed while wiring this up, both of which would have destroyed
a real run: the wrappers wrote results under /workspace, which is NOT a mounted path
(only -v ${LOG_PATH}:/run_logs is), so ten runs of results would have gone to the
container overlay and died with the container; and the pooler globs were hardcoded
to the default scenario labels, so overriding SCENARIOS would have matched nothing
and said so only at the very end of a ten-run job.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…orting Two defects in the existing NIAH sweep, both of which produce a misleading result rather than an error. NIAH_WORDS measures WORDS. The context numbers we are asked about are TOKENS, and for this filler the ratio is ~1.30 -- so "950,000" read as words is ~1.24M tokens, ABOVE a 1,048,576 window. vLLM REJECTS an over-length request with a 400 rather than truncating, and that arrives at the client as a transport error indistinguishable from a dead server. NIAH_TOKENS calibrates against the real tokenizer and approaches the target FROM BELOW: overshooting a window is a hard failure, undershooting is a slightly shorter test. Without transformers installed it falls back to NIAH_TOKENS_PER_WORD and prints an UNCALIBRATED warning, so the lengths are never silently passed off as exact. A bare score was uninterpretable. "7/10 at 950K" and "7/10 at 950K, all three misses past 80% depth" call for completely different investigations -- the second says the tail of the context is being dropped. Misses now report their depth, with a decile histogram. Exit codes 3 and 4 are deliberately distinct: 3 = some rung produced no usable result (dead server, timeout, rejection), 4 = every rung scored but a mean fell below NIAH_MIN_SCORE. "We could not measure it" and "we measured it and it is bad" are not the same finding. benchmark_niah_long.sh drives the ladder and is where the wall-clock budget lives. Prefill is ~QUADRATIC here: the sparse indexer scores each 8,192-token prefill chunk against all preceding keys (index_topk=2048 caps the DECODE key set, not the prefill scan). Scaling from a measured 4.80 s TTFT at 28,672 tokens the default ladder is ~3.3 h at one seed with no warmup, ~6.6 h with warmup, ~13.2 h at three seeds with warmup. The last TWO rungs are 74% of that, so dropping 950,000 alone buys back 44% -- the per-rung table is in the header so that trade can be made rather than rediscovered. Hence NIAH_SEEDS=0 and NIAH_WARMUP=0 here, unlike benchmark_niah.py. Timeouts scale per rung, quadratically, floored at 300 s. A single flat timeout sized for 950K means a dead server burns 4.4 h on the 32K rung and the job wall clock expires before a single line prints. Readiness is a real completion, not /v1/models: that endpoint is wrong in BOTH directions on this router -- 200 while every request 503s, and 503 while completions succeed -- because its listing path and its forwarding path consult different state. And the ladder invokes benchmark_niah.py directly rather than through benchmark_niah.sh, whose trailing `| tee` always exits 0, so a failing ladder would have reported success. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t them Registers niah_long, avg_80k_ten and avg_200k_ten as BENCHMARK_SCRIPT values, adds the five new files to REQUIRED_FILES, and fixes two latent launcher defects found while wiring them up. The docker -e passthrough block is a hard gate: an env var not listed there simply does not exist inside the container. The ENTIRE customer_slo family -- SCENARIOS, SLO_ITERS, SLO_TTFT_MS, SLO_TPOT_MS, WORKLOAD_MODE, TAIL_FRAC, PREFIX_FRAC, MAX_PROMPTS, BURSTINESS, RESAMPLE_PER_ITER, SEED_BASE, DP_RANKS, TGT_* -- was never in it. That benchmark had only ever been runnable at its compiled-in defaults, and every override was accepted at the sbatch line and silently dropped. Adds those plus the four new NIAH token knobs. The selector's error message listed a stale set of valid values, omitting niah and niah_perf. A correct BENCHMARK_SCRIPT=niah_perf that tripped over an unrelated typo would be told its own value was illegal, which sends you looking in the wrong place. BENCHMARKS.md documents the three benchmarks for use against ANY OpenAI-compatible server, not just this launcher, since the question they answer is not specific to this cluster. It carries the dependency matrix (benchmark_niah.py, pool_workload.py and slo_report.py are stdlib-only and run anywhere; gen_workload.py needs transformers; only the SLO driver needs vllm bench serve), the runtime budget, the env-var tables, the exit-code contract, and the failure modes that present as good results -- chiefly that a rejected over-length request vanishes from the latency statistics and shows up only as a lower `completed` count. README.MD documented `export BENCHMARK_SCRIPT_FILE=benchmark_niah.sh`. That is the exact form the launcher's `case` discards: it assigns BENCHMARK_SCRIPT_FILE with a plain `=`, so an inherited value is overwritten and the default throughput sweep runs instead. Corrected to BENCHMARK_SCRIPT=niah, with a note, since a documented-but-inert knob costs a full job to diagnose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
argv_assert.sh grepped the slurm for the literal
CONNECTOR_ENV_FILE="${SCRIPT_DIR}/connectors/${CONNECTOR}.env"
but 24b5c9c made that path overridable for site portability:
CONNECTOR_ENV_FILE="${CONNECTOR_ENV_FILE:-${SCRIPT_DIR}/connectors/${CONNECTOR}.env}"
so the suite has been red since, on code that is correct. A test that fails when the
product is right is worse than no test: it trains you to ignore the suite, and the
next real regression lands in a run that was already failing.
Matches the per-connector default instead of the exact assignment, which is the
behaviour actually under test. 25/25 pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mage
Adds a self-contained benchmark folder alongside scripts/vllm_dissag, following
the scripts/kvcache_transfer_bench layout (own Dockerfile in-dir, own README,
not registered in models.yaml).
What it measures: MoRI-IO transfer bandwidth/latency between two MI355X nodes
across a geometric ladder of block sizes, engaging all 8 GPU-data-plane RDMA
rails (Pensando/ionic, 400 Gb/s each) at once. benchmark.py prints one table per
rail; aggregate_sweep.py folds the 8 into a fabric view. It sums bandwidth but
takes the MAX of latency, because a batched handoff completes when its slowest
rail completes -- and it prints per-rail min/max, since a lame rail is invisible
in a sum and is exactly what a prefill->decode handoff waits on.
Measured (job 5642, exit=0, logs in results/):
- peak 378.1 GB/s aggregate avg at 4 MiB = 94.5% of 400 GB/s nominal; best
single rail 47.43 GB/s = 94.9% of its own line rate, with RoCE headers,
ICRC and MoRI chunk descriptors counted against it.
- the 4 MiB knee was predicted from PlanChunkGeometry before the run and
confirmed by it: softCount = min(ceil(total/chunkBytes), maxChunks) with the
build defaults 65536 x 64 saturates at exactly 4 MiB. The 32 MiB row is
64 x 512 KiB chunks, NOT "MoRI is slower at 32 MiB".
- rail spread collapses 30.7% (4 KiB) -> 0.5% (2 MiB): all 8 rails healthy.
Consequence for GLM-5.2 disagg: a 28,672-token prefill moves ~1.2 GiB of KV,
~3.4 ms at this bandwidth, 0.05% of a 7 s TTFT budget. KV transport is not and
will not become the TTFT constraint here. This closes a hypothesis rather than
opening one -- it says don't spend time tuning MoRI-IO block sizes.
Provenance, which is the point of committing the raw logs: the measured engine
is the workload image's own pinned libmori (MORI_REF=42e895472b08), verified by
importing it in-container. The bind-mounted source cannot shadow it (the repo's
python package lives at python/mori, not the tree root) and exists solely to
supply tests/python/io/benchmark.py, which the wheel does not install. So the
number characterises the stack this PR actually ships.
The mounted harness was an older 6ad812c checkout whose benchmark.py predates
--mem-type/--max-chunks/--chunk-bytes, so those knobs ran at their defaults and
CHUNK_SWEEP=1 was skipped loudly. That is a harness limitation, not an engine
one: the pinned 42e8954 accepts all three. The Dockerfile here bakes a harness
at the image's own MORI_REF -- and fails the build if the two disagree -- so the
4 MiB knee can be isolated, not merely explained. It also installs numactl,
absent from the workload image, which left MatchCpuNics() ordering unpinned.
results/.gitignore un-ignores *.log/*.json for this directory only. The root
rule is right in general; here the logs are the deliverable, and a result table
whose raw evidence is not in the tree cannot be re-checked later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5 tasks
…ant bring-up MODEL_CONFIG_PREFILL/MODEL_CONFIG_DECODE are composed from models.yaml alone (vllm_disagg.sh:200-223), so per-role flags such as --gpu-memory-utilization and --max-model-len cannot be injected at submit time -- only the protect-listed env keys can. vllm_disagg.sh:161 already honours a MODELS_YAML override, but the variable was absent from the -e passthrough, so it never reached the container and the override silently did nothing. Pass it through. Because $NIXL_REPO_DIR is bind-mounted at /opt/nixl-vllm-cookbook (run_xPyD_models.slurm:706), an overlay copy placed beside models.yaml is now selectable with MODELS_YAML=/opt/nixl-vllm-cookbook/<file>.yaml, which keeps transient cluster workarounds out of the tracked recipe. GLM52_MI355X.md section 4 gains three bring-up notes learned the hard way on a shared node: the overlay mechanism above (a host path silently fails); --gpu-memory-utilization is a fraction of TOTAL card capacity and cannot shrink the weights floor, so with a co-tenant the KV budget can go negative while the startup check still demands free >= util * total -- unsatisfiable, wait rather than tune; and the AITER JIT baton, where a ~5 min hipcc compile and a stale 0-byte lock look identical, the cache is per-hostname, and deleting the build dir under live workers turns a wait into a hard import failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…de pair TPOT 33.2 / 34.7 / 38.6 ms at concurrency 16/32/64 (ISL/OSL 28672/1024, EP8), matching the earlier 33.7 / 36.4 / 40.2 within noise on different hardware and after a co-tenant episode. Zero failed requests. Adds the TTFT and throughput columns, since TTFT -- not TPOT -- is the axis that misses the SLO. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
benchmark_niah_256k.sh runs the 8k/32k/64k/131k/262k retrieval ladder with 3 seeds + warmup and an end-to-end readiness probe. run_xPyD_models.slurm gains the niah_256k dispatch case and the REQUIRED_FILES entry so preflight passes. Co-Authored-By: Claude <noreply@anthropic.com>
benchmark_customer_slo.sh gains a per-cell warmup rehearsal (CELL_WARMUP) and a [STALL] guard that records timed-out cells instead of hanging silently. benchmark_avg_80K_ten.sh sets the 256k-ctx/80K scenario defaults and a prompt-count banner. slo_report.py --min-iter drops warmup iterations from the reported average. Co-Authored-By: Claude <noreply@anthropic.com>
…-IO findings GLM52_MI355X.md documents the weight download (zai-org/GLM-5.2-FP8, amd/GLM-5.2-MXFP4), the reproduced throughput sweep, the util-overlay caveat, the NIaH-to-262K reproducibility caveat, and the MoRI-IO lost-KV-notification finding that stalled 3 of 4 avg-80K cells. Dockerfile changes are comment-only (built image is bit-identical) -- correcting stale gfx942/arch and pin-set text. Co-Authored-By: Claude <noreply@anthropic.com>
…-node runs MORIIO_JOBID2 lets the sweep borrow an already-allocated second node via --overlap instead of requiring a single 2-node allocation. Co-Authored-By: Claude <noreply@anthropic.com>
…16 on ionic EP16 shards experts across 2 nodes, so the MoE all2all crosses the ionic fabric; GPU-initiated IBGDA doorbell MMIO fails under KVM/VFIO on ionic. Tej's PR #558 (itej89/mori @ feat/ep-rdma-sharing) adds a host-CPU proxy (MORI_EP_OVER_RDMA=1 -> TransportType::PROXY) that rings the NIC doorbell via ibv_post_send. Add --build-arg WITH_MORI_EP_OVER_RDMA=1 (default 0) to switch the MoRI source to Tej's fork and apply docker/mori_pr558_ionic.patch (ionic atomic-MR strip: MaybeAddRelaxedOrderingFlag drops IBV_ACCESS_REMOTE_ATOMIC when MORI_IO_DISABLE_ATOMIC_MR=1, since ionic rejects atomic-capable MRs with EINVAL; plus an off-by-default MORI_PROXY_DEBUG trace). Default build (=0) reproduces the validated 1P/1D EP8 image byte-for-byte (ROCm/mori @ 42e895472b08), unchanged.
Add the EP16 (DP16 across 2 nodes per role) disaggregated recipe on the Crusoe MI355X + AINIC (ionic) cluster, driven through vllm-router. This is a different cluster/shape than the AAC 1P/1D-EP8 sweep in GLM52_MI355X.md -- kept separate, not merged. - vllm_pd_ep16_launch.sh / ep16_orchestrate.sh: per-node launcher + 4-node driver, handling the four bring-up gotchas (master must not pass --data-parallel-start-rank; NCCL on TCP via NCCL_IB_DISABLE=1 for cross-node DP; 16G MoRI heap vs util budget; GPU-drain before relaunch). - GLM52_EP16_IONIC.md: the EP16 recipe, why it needs PR#558, and the run steps. - RESULTS_CRUSOE_IONIC.md + glm52_crusoe_ionic_report.html: three-config matrix. TPOT ladder TP8 ~22ms / EP8 ~60ms / EP16 ~117ms (each all2all hop ~doubles it); EP8 scales to 371 tok/s @ con64; NIAH 10/10 for all three. - Documents blocked levers: DBO (MoRI supports_async=False), MXFP4 (Triton Code-209 on gfx950), MTP (hangs in disagg).
raviguptaamd
requested review from
Cemberk,
itej89,
lcskrishna and
leconcio
as code owners
August 27, 2026 08:05
Contributor
Author
EP16 cross-node disagg + Crusoe/ionic results (additive)Adds the EP16 (expert-parallel across 2 nodes per role) path and a three-config comparison, without touching the validated AAC 1P/1D-EP8 content or the NIAH harness (already correct). 1. MoRI PR#558 for cross-node EP on ionic (
2. EP16 recipe + Crusoe/ionic results (
Results (Crusoe MI355X + ionic, GLM-5.2-FP8, via vllm-router — separate cluster/shape from the AAC EP8 sweep, not merged):
Each all2all hop ~doubles TPOT. NIAH 10/10 for all three (2K/20K/50K/90K). Blocked levers documented: DBO (MoRI MXFP4 fix across all cases is tracked as follow-up. |
…rrent stack
The earlier Triton Code-209 ("no kernel image for gfx950") in the aiter MXFP4 quant
kernel is resolved on the current aiter/vLLM/flydsl stack; the MXFP4 MoE kernels now
compile for gfx950. Verified standalone TP8 (--quantization quark): Application startup
complete, 0 kernel errors, coherent smoke output. Model load 51.7 GiB/rank (~42% smaller
than FP8's ~89 GiB, as expected for Quark MoE-weights-only). Supersedes the "config-only,
never booted" note. Disaggregated 1P/1D MXFP4 + perf sweep remain as bring-up.
Decision guide grounded in the collectives. Documents the measured throughput crossover (EP8 overtakes TP8 at ~con48 @ 8K, ~con16 @ 32K; TP8 flatlines at ~69 tok/s @ 32K while EP8 scales), and explains it from the per-step collective each topology runs: TP8 allreduce + fragmented 1/8-size MoE GEMMs (memory-bound, plateaus) vs EP8/16 dispatch+combine all2all + full-size expert GEMMs (scales). Cost model TPOT ~= compute_floor + bytes/token / link_BW reproduces the 22/60/117 ms ladder from measured XGMI (~130 GB/s) vs cross-node ionic RDMA (~37 GB/s) bandwidths. Decision matrix by (context, concurrency, SLA): TP8 = latency tier, EP8 = throughput tier, EP16 = capacity/scale-ceiling tier. Notes TP8-128K (node-blocked) + EP16-128K-con64 as open cells.
…figs)
Self-driving driver for the GLM-5.2 long-context study. For one live config
(TP8/EP8/EP16) it walks {50K,100K,200K,300K,500K,750K} x con{8,16,32,64},
gating each size on NIAH retrieval (token-calibrated, fixed harness maxtok=256
enable_thinking:false) BEFORE trusting its perf numbers, and records every cell
-- success OR capacity-ceiling failure (OOM/KV/ctx-overflow/stall) -- to a CSV
with full provenance (config,size,con,node,image,util,timestamps) plus a live
append to the NFS logbook. Ceiling-aware: skips higher con once a size OOMs.
The capacity ceilings ARE the result: where TP8/EP8 stop and EP16 survives is
the measured per-config advantage. Reuses benchmark_niah_v2.py + vllm bench serve.
Upstream every source fix needed for MTP (speculative decode) on disaggregated serving across TP8 / EP8 / EP16, so PR ROCm#205 is reproducible on any cluster (no hand-patching). - apply_all_patches.sh: idempotent master applier; auto-detects vllm/aiter install dirs, runs all anchor-based patches in safe order, hard-fails on a changed anchor (so a vLLM bump can't silently drop a fix). - Two previously-missing gpu_worker patches (EP16-MTP boot deadlocks): * apply_glm_vllm_skip_profile_run_fix.py (VLLM_SKIP_PROFILE_RUN): skip the determine_available_memory profile forward whose MoE all2all deadlocks cross-node at DP16. * apply_glm_vllm_skip_warmup_dummy_fix.py (VLLM_SKIP_WARMUP_DUMMY): gate the compile_or_warm_up_model elif _dummy_run that drives the MTP speculator into the same DP all_reduce. - README_MTP_DISAGG.md: per-mode env matrix, apply flow, honest perf note (only DECODE_CUDAGRAPH_MODE=NONE costs perf and is being removed), the never-partial-restart operational rule, and cudagraph in-progress status. All gates default to stock behavior; EP16-specific gates are no-ops on TP8 (dp_size=1) and EP8 (single-node DP8), so non-MTP runs are unaffected. Co-Authored-By: Claude <noreply@anthropic.com>
…drop to eager The eager DECODE_CUDAGRAPH_MODE=NONE fallback was a workaround, not the target: it costs per-step kernel-launch overhead. Root cause of the capture deadlock: cudagraph capture_model() runs _dummy_run at each size through a DP-group all_reduce; ranks capture at their own pace -> asymmetric arrival -> deadlock. Two patches keep cudagraphs ON instead of disabling them: - apply_glm_vllm_startup_dp_uniform_fix.py (VLLM_STARTUP_DP_UNIFORM): in sync_cudagraph_and_dp_padding, fill the DP coordination tensor locally (all ranks pass the same dummy shape at startup, so the all_reduce result is uniform anyway) -> graphs capture without the deadlocking collective. - apply_glm_vllm_startup_dp_uniform_worker_fix.py (VLLM_STARTUP_DP_UNIFORM_ENABLE): set that flag ONLY around warmup+capture (try/finally) so runtime inference keeps the real cross-rank all_reduce. Wired into apply_all_patches.sh; README §4 documents the graphs-ON env recipe. Both gates default off (no effect on TP8/EP8/non-MTP). Live graphs-ON TPOT A/B is the remaining validation item. Co-Authored-By: Claude <noreply@anthropic.com>
bench_cell referenced lowercase $osl (unset) instead of $OSL for the timeout math and the --random-output-len arg; point NIAH_PY at v3. Co-Authored-By: Claude <noreply@anthropic.com>
Image is now self-contained (patches compiled in, no runtime bind-mounts):
- VLLM_REF -> raviguptaamd/vllm@glm5.2-mi355x-mtp-ep16 (623fdc946 + 8 MTP/
EP16 stability commits: MTP KV block-fix; VLLM_SKIP_{FWDCTX_DP_AR,
DP_SYNC_ON_PROFILE,PROFILE_RUN,WARMUP_DUMMY} startup DP-deadlock gates;
DSA kernel/indexer/persistent)
- AITER_REPO/REF -> raviguptaamd/aiter@glm5.2-mi355x-mtp-ep16 (e03fa6040 +
AITER_FORCE_CK_FMOE gate for EP16-MTP 1tg->CK)
All gates default to stock; no-op on TP8/EP8/non-MTP. Measured: TP8-MTP
-41%, EP8-MTP -38% TPOT; EP16-MTP boots+serves; NIAH 10/10.
README_MTP_DISAGG: build-from-forks TL;DR; apply_all_patches.sh retained
for dev/other-base regen (source of truth for the fork commits). Cudagraph
patches (VLLM_STARTUP_DP_UNIFORM*) stay apply-script-only (WIP).
Co-Authored-By: Claude <noreply@anthropic.com>
MoRI (itej89/mori PR#558) uses setuptools_scm, which fails to parse the reachable git tag 'v7-verified' (not PEP440) -> "Can't parse version from tag" -> wheel build aborts. Pin SETUPTOOLS_SCM_PRETEND_VERSION=0.1.0 on the MoRI pip install so the version is fixed and tag parsing is bypassed. Surfaced building the fork-native image from a clean context. Co-Authored-By: Claude <noreply@anthropic.com>
Some base images ship libibverbs 1.16 (provider ABI rdmav59) as the default .so.1, but the host AINIC ionic driver is rdmav34 (libibverbs 1.14). MoRI compiled against 1.16 aborts at runtime with "no transport available for peer" (context.cpp:353) because libionic-rdmav59.so does not exist. Repoint libibverbs.so.1 -> 1.14.39.0 before the MoRI compile so MoRI both builds and runs against 1.14, matching the host driver -- no runtime lib mounts, one consistent ABI. Matches the old validated image (.so.1 -> 1.14.39.0). Surfaced deploying the fork-native image on the ionic cluster. Co-Authored-By: Claude <noreply@anthropic.com>
The pre-MoRI pin ensures MoRI compiles against 1.14, but later layers reset the .so.1 symlink back to 1.16/rdmav59 (base packages / apt / pip). Add a final re-pin as the last layer so the shipped image RUNS against 1.14, matching the host ionic driver. Verified: image symlink now libibverbs.so.1.14.39.0. Co-Authored-By: Claude <noreply@anthropic.com>
…working EP16 + MTP + decode cudagraph does not serve (proven cross-node cudagraph capture-lockstep; ~10 runs, py-spy). Prior docs overstated it as "boots + serves" / "one stall away". Correct all 5 MTP docs to state EP16-MTP is a documented known limitation with the real root cause and the fix path (cross-rank barrier in vLLM's capture loop, scoped not implemented). Also update EP8-MTP number to measured -44% (33.2 vs 58.9 ms). Working MTP: TP8 (-41%), EP8 (-44%); EP16-noMTP serves (114.9 ms cudagraph).
ppalaniappan-amd
removed their request for review
September 1, 2026 16:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables GLM-5.2 (FP8 + MXFP4) on MI355X (gfx950) with AMD AI NIC (ionic) for 1P/1D EP8
disaggregated serving over MoRI-IO. Measured on AAC, 2 nodes x 8 x MI355X, ROCm 7.2.3.
Result
TPOT target met. ISL/OSL 28,672/1,024, FP8, EP8, 1P/1D:
Target is 50 ms avg. Before this change it was 104 ms.
TTFT target not met, and this PR does not fix it. Prefill measures 6,135–8,359
tok/s/rank against a 34,000 target (~4–5x short), and the fitted single-request TTFT at
80K ISL is ~13.7 s against a <7 s SLO.
GLM52_MI355X.md§5 records the profile behind that(MLA sparse attention ~82% of prefill, all GEMM 17.8%, MoE 2.9%), the six levers already
closed by measurement so they are not re-litigated, and why PCP is not the near-term answer.
The two load-bearing values
DECODE_CUDAGRAPH_MODE: FULL_AND_PIECEWISE— the single change that moved TPOT104 ms -> 34–40 ms.
PIECEWISEsplits the decode graph on the 3 DSA ops x 78 layers,~234 launch boundaries per step. That cost is batch-independent, so it presents as a
latency floor that does not move with concurrency, batch size, or fabric — which is why it
survived so much tuning before being found. Co-requisite:
use_inductor_graph_partitionmust stay ON and no bare
--enforce-eager, or boot and warmup both pass and the first realMLA decode dies.
prefill.dp: --gpu-memory-utilization 0.72— the MLA chunked-prefill workspace is sizedfrom a hardcoded 64k-token clamp (
determine_chunked_prefill_workspace_size), not frommax_num_batched_tokens, so it allocates 3.50 GiB on top of the KV pool regardless ofscheduler batch size — lazily, on the first long prefill, i.e. after boot and warmup have
both reported healthy. 0.80 OOM'd.
A silent wrong-arch bug likely affecting other Dockerfiles here
The base image
rocm/vllm-dev:ci_base-*exportsPYTORCH_ROCM_ARCHandMAX_JOBSasENV, which shadows the same-named
ARGin the child Dockerfile. SoARG PYTORCH_ROCM_ARCH="gfx950"was silently inert and the build produced gfx942 kernelswhile appearing to succeed. Fixed by renaming to
BUILD_ROCM_ARCH/BUILD_MAX_JOBSandassigning through. Caught only because the base's
MAX_JOBShappened to be empty(
int('')crash) — otherwise it fails silently, so it is worth checking other childDockerfiles in this tree for the same shadowing.
Also in here
connectors/moriio.env.aac— AAC/ionic platform env: 8 railsrocep{9,25,105,121,137,153,233,249}s0,MORI_IB_GID_INDEX=1, mgmt NIC for bootstrap.FABRIC_SUBNETdefaulted to the MI300X cluster's10.158.prefix; on AAC (
10.2.80.) that advertises aMASTER_ADDRthe peer cannot reach and therun hangs at
Waiting for nodesforever with no error. Unlike every other key in the envfile this one is consumed on the host, so the
.envloader needed an explicit one-keyhost-export allow-list — a plain env entry would have been inert.
parse_to_csv.py— was silently dropping the first (lowest-concurrency) cell of everyrun (21
[RUNNING]cells -> 20 rows), tagging every MoRIIO run asnixl, and filing everyMI355X run under
gfx942. All three fixed; dropped rows now warn instead of vanishing.benchmark_niah.py— the accuracy check printed a score but always exited 0; now areal pass/fail gate, with jittered needle placement.
GLM_PERSIST_GATE,GLM_DSA_SENTINEL_FIX):both are correct on older images but here cause a C++ abort at
asm_mla.cu:945and ahipErrorIllegalAddressrespectively. The gate is by image, not by model..nfs0000000016f44d2b00008188, a stalebyte-copy of
run_xPyD_models.slurm).Testing status
FP8 is the measured path — NIAH accuracy plus the 28,672/1,024 sweep at concurrency
16/32/64 above.
GLM-5.2-MXFP4is config-only and has not been booted yet — bring-up is queued. Itsrecipe is the FP8 one with the two values that provably do not transfer re-derived. Both
models.yamlandGLM52_MI355X.mdflag it as untested; treat its first run as a bring-up,not a benchmark.