Add DeepSeek-V4-Flash-FP8 (MoRI-EP wide-EP, EP8 + EP16) to sglang_disagg - #240
Open
raviguptaamd wants to merge 10 commits into
Open
raviguptaamd wants to merge 10 commits into
raviguptaamd wants to merge 10 commits into
Conversation
raviguptaamd
requested review from
Cemberk,
amathews-amd,
coketaste,
gargrahul,
itej89,
lcskrishna and
leconcio
as code owners
August 29, 2026 05:36
Contributor
Author
Reproduction artifacts addedSelf-contained perf report + repro scripts are now in
Anyone can re-measure with: pull |
Adds DeepSeek-V4-Flash-FP8 as a MoRI-EP disaggregated model: - models.yaml: DSV4-Flash recipe (dsv4 attention, fp8_e4m3 KV, sparse-MLA, DP-attention mandatory, MoRI-EP with --deepep-mode normal (prefill/HT) / low_latency (decode/LL), eager prefill) - models.json: EP8 1P1D and EP16 2P2D runnable entries - allowlists (mori_io_ep + slurm): register DeepSeek-V4-Flash-FP8 - set_env_vars.sh: DSV4 load-bearing env (MODEL_NAME-guarded) Framework robustness fixes (generalize to non-SLURM / non-Mellanox / bnxt): - Persistent JIT cache mount + orphaned-lock sweep (MoRI/aiter FileBaton has no PID/timeout; a lock orphaned by a killed run otherwise hangs later runs) - Router-ready gate: poll peer /health over the network (FS-agnostic) instead of grepping a node-local peer log, so it works without a shared filesystem - host_ip / socket-iface autodetect: use the route dev field and match the IPADDRS scope (fabric), not the default-route metric - set_env_vars: don't append mlx5_* (Mellanox-only) to the NIC list - Expose host libbnxt_re + ldconfig in-container for the bnxt (USE_CX7_NICS=0) RoCE path so MoRI finds a working RDMA transport - KEEP_ALIVE=1 to leave servers running for external NIAH/perf testing Validated on MI308X/gfx942 + Broadcom Thor2: NIAH 18/18 (1K-200K, depths 10/50/90%) on both EP8 1P1D and EP16 2P2D; perf at 8K/1K and 16K/1K @ con 16/32. Prebuilt image: rocmshared/sglang-disagg-dsv4:mori-mi308-pr Co-Authored-By: Claude <noreply@anthropic.com>
Self-contained artifacts to reproduce/re-measure DeepSeek-V4-Flash-FP8 EP8/EP16 results: - PERF_REPORT.html : NIAH (18/18 each) + perf tables, image digest, full reproduce steps - run_dsv4_nonslurm.sh : per-node docker launch driver (non-SLURM / bnxt) - niah.py : needle-in-a-haystack harness (ENDPOINT env, defaults localhost) - perf.py : lightweight concurrent-request throughput timer No cluster-internal IPs/hostnames; endpoints are env-parameterized. Co-Authored-By: Claude <noreply@anthropic.com>
…ments Replace an internal cluster codename in two comment lines with a neutral "bnxt/Thor2 (Broadcom)" / "fabric subnet" description. No behavior change. Co-Authored-By: Claude <noreply@anthropic.com>
raviguptaamd
force-pushed
the
add-deepseek-v4-flash-sglang-disagg
branch
from
August 29, 2026 06:25
a0b6819 to
2cbb7b8
Compare
DeepSeek-V4-Flash decode previously ran eager (--disable-cuda-graph) because
MoRI-LL crashed under CUDA-graph capture. Root-caused to two independent issues,
both fixed here:
1. MoRI too old. Pinned 158c7e83 (2026-06-08) predates the fixes for the exact
capture crashes:
- #363 "guard dispatch kernels against out-of-range expert id" -> the
low_latency_async.cpp:360 `(pe>=0 && pe<worldSize)` device assertion.
- #505 "AsyncLL slot double-alloc when top-k does not divide warpSize" ->
DSV4-Flash is topk6, warpSize64 (6 does not divide 64).
Bump MORI_COMMIT -> 7c51d18f (includes both, plus recent [AMD][DSV4] fixes).
2. Recv-buffer sizing conflated with the dispatch-token cap. The launcher set
SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK (which sizes the LL recv buffer,
~= PER_RANK * world_size) equal to MORI_MAX_DISPATCH_TOKENS_DECODE. Shrinking
the dispatch cap then starved the recv buffer -> low_latency_async.cpp:324
"Total recv token overflow" during capture. Decouple them (CI sets them
separately); default decode PER_RANK to 512. Keep this role-specific: prefill
needs PER_RANK >= chunked_prefill (8192), so set_env_vars must NOT export it
globally.
models.yaml: decode.dp now captures a bounded set (--cuda-graph-bs 1..64,
backend=full); prefill stays eager (MoRI-HT). set_env_vars adds the decode
dispatch caps (guarded by MODEL_NAME).
Validated on MI308X/gfx942 + Thor2, decode cuda graph=True, 8K/1K:
EP8 1P1D : con16 35->57 tok/s (+63%), con32 ->104; NIAH 18/18 (1K-200K).
EP16 2P2D: con16 62->166 tok/s (+168%), con32 65->192 (+195%).
Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
Decode CUDA-graph now enabled (was eager) — major perf gainFollow-up to the earlier "decode runs eager" caveat: decode now runs under CUDA-graph (MoRI-LL, backend=full). Root-caused the capture crash to two independent issues, both fixed in commit
Perf (8K/1K, decode eager → cudagraph):
Correctness unchanged: EP8 NIAH 18/18 (1K–200K) with cudagraph, Updated image: |
- prefill chunked-prefill-size 8192 -> 131072. Under DP-attention SGLang divides the chunk by dp_size (server_args.py: chunked_prefill_size //= dp_size), so 8192 became 512/rank and serialized the 100K prefill. 131072/dp16 = 8192/rank. - launcher: set the prefill-branch MoRI recv-buffer PER_RANK (SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK, default 16384) so it satisfies "PER_RANK >= per-rank prefill dispatch (chunked/ dp_size)" with the larger chunk. - decode: --num-continuous-decode-steps 2 amortizes per-step scheduling/MoRI-LL dispatch overhead -> TPOT p95 55->49-50ms (meets <50ms at con24/36). Measured EP16 2P2D, 100K/1.1K, MI308X: TPOT p50 ~48ms / p95 49-50ms at con24/36. TTFT for 100K remains compute-bound (~82s) on ROCm because the sparse-MLA prefill kernel is CUDA-only (SGLang disables SGLANG_OPT_FLASHMLA_SPARSE_PREFILL on HIP for DSV4) and prefill-CP requires interleave which conflicts with the mandatory DP-attention; <10s TTFT holds for ISL up to ~16K. These are upstream enablement gaps, tracked separately. Co-Authored-By: Claude <noreply@anthropic.com>
…aph + 100K/1.1K SLO The HTML report was previously skipped by .gitignore (*.html), so the PR referenced a file that was never committed. Force-add it and refresh with the current state: decode CUDA-graph enabled (MoRI 7c51d18f), image pr2, full customer SLO sweep (100K/1.1K con 12/24/36 TTFT/TPOT percentiles), TPOT p95<50ms met at con24/36, TTFT-at-100K root cause + upstream ask. Co-Authored-By: Claude <noreply@anthropic.com>
… accuracy identical DeepSeek-V4-Flash ships one MTP head (num_nextn_predict_layers=1). Enabling it as the EAGLE draft (--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4) accepts ~3 tokens per full-model forward pass, cutting decode latency with NO accuracy change (every speculative token is verified by the full model → bit-identical output). Measured EP16 2P2D, 100K/1.1K, MI308X (vs decode-cudagraph baseline): con12 TPOT p95 59->39ms, con24 49->40ms, con36 50->39ms; throughput +5-14%. NIAH 18/18 unchanged. accept len ~3, cuda graph: True (MTP + MoRI-LL coexist; no paged_mqa_logits_metadata assertion on SGLang 0.5.15 / gfx942). Supersedes --num-continuous-decode-steps 2 (MTP gives a larger TPOT win). Adds slo_harness.py (streaming TTFT/TPOT percentiles; counts usage.completion_tokens so per-token latency is correct under speculative burst delivery). Co-Authored-By: Claude <noreply@anthropic.com>
…y identical) Update PERF_REPORT with the MTP results: full 100K/1.1K customer sweep, the eager->cudagraph->MTP optimization path, and the accuracy-preserved note. Co-Authored-By: Claude <noreply@anthropic.com>
…ated Add §4.3 MTP: EAGLE spec-decode enabled on both topologies, NIAH 18/18 each (EP8 paced), accept len ~3 (EP16) / ~2 (EP8), TPOT p95 to 39-40ms (EP16) / ~54ms (EP8), accuracy identical. Note EP8 single-node burst sensitivity at extreme context; EP16 recommended for 100K/1.1K. MTP works with MoRI-LL + decode CUDA-graph on 0.5.15/gfx942 (no paged_mqa assertion). Co-Authored-By: Claude <noreply@anthropic.com>
…I ROCtx) Adds steps_to_profile.md — field notes for profiling MoRI-IO + MoRI-EP with rocprofv3 on a stock DSV4 MoRI image (no MAD-private profiling plumbing), taken on MI308X + Broadcom Thor2. Documents what works (MoRI ROCtx markers PR #510 already in image; rocprofv3 follows forks; bnxt NIC counters) and — importantly — what does NOT: wrapping all 8 DP workers under rocprofv3 --kernel-trace crashes the gloo collective on MI308X (per-kernel hook overhead stalls a sync). --dist-timeout and -P collection-period delay but do not remove the overhead; --attach cannot retroactively instrument kernels. Recommends rank-0-only (needs a small SGLang patch), marker-trace-only, or a TP-only serve. Also wires a RUN_PROFILE=1 gate into sglang_disagg_mori_io_ep.sh (_prof_prefix: rocprofv3 wrapper with -P + MoRI ROCtx gates + PROFILE_KERNEL toggle; raised dist-timeout). Marked in-code as the BASE to build the rank-0 patch on, not a working full-DP capture — see steps_to_profile.md §3/§4. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds DeepSeek-V4-Flash-FP8 as a MoRI-EP disaggregated model under
scripts/sglang_disagg, with both EP8 (1P1D) and EP16 (2P2D) topologies. Mirrors the existing DeepSeek-V3 MoRI-EP entry, plus DSV4-specific handling (dedicateddsv4attention backend, fp8_e4m3 KV, sparse-MLA, mandatory DP-attention, MoRI-EP HT-prefill / LL-decode split).What's added
models.yaml—DeepSeek-V4-Flash-FP8recipe.num_key_value_heads=1→ DP-attention is mandatory;--deepep-mode normal(prefill/HT) +low_latency(decode/LL); eager prefill.models.json— two runnable entries:..._ep8_1p1d(xP=1 yD=1) and..._ep16_2p2d(xP=2 yD=2), bothDP_MODE=1.sglang_disagg_mori_io_ep.sh+run_xPyD_models.slurm) — register the model.set_env_vars.sh— DSV4 load-bearing env,MODEL_NAME-guarded.README.MD— model section, image, and validation.Framework robustness fixes
These generalize the framework beyond SLURM + Mellanox (validated on a non-SLURM Broadcom Thor2 / bnxt cluster); each is guarded so existing setups are unaffected:
FileBatonhas no PID/timeout, so a lock orphaned by a killed run hangs every later run. Sweep stale locks pre-launch + mount a persistent, image-versioned JIT cache (compile-once)./healthpoll — instead of grepping a node-local peer log (invisible without a shared FS). FS-agnostic; falls back to the log grep.host_ip/ socket-iface autodetect — use the routedevfield and match theIPADDRSscope (fabric), not the default-route metric.mlx5_*(Mellanox-only) to the NIC list inset_env_vars.sh.USE_CX7_NICS=0) — bind-mount the host's ABI-correctlibbnxt_re+ldconfigin-container so MoRI finds an RDMA transport.KEEP_ALIVE=1— leave servers running for external NIAH/perf testing.Validation (MI308X / gfx942 + Broadcom Thor2)
Served via
sglang_router, greedy/v1/completions.Correctness — needle-in-a-haystack, needle at 10/50/90% depth:
Factual length-sweep coherent through 200K on both.
Perf (functional; eager decode, all requests succeeded):
Image
docker pull rocmshared/sglang-disagg-dsv4:mori-mi308-pr(SGLang + MoRI on ROCm 7.2, gfx942), or build fromdocker/sglang_disagg_inference.ubuntu.amd.Dockerfile.Known follow-ups
--disable-cuda-graph) — the MoRI-EP low-latency kernel HIP-launch-fails under cudagraph capture on the pinned MoRI commit. Re-enabling decode cudagraph + SLO perf tuning is a follow-up; correctness is unaffected.Test plan
🤖 Generated with Claude Code