Conversation
Adds scripts/vllm/kimik3_mi300x/ with three self-contained colocated vLLM serving recipes for Kimi-K3 (MXFP4) on AMD Instinct MI300X / gfx942, complementing MAD's existing single-node gfx950/MI355X K3 recipe. MI300X has 192 GB/GPU, so the ~1.5 TB K3 checkpoint does not fit a single 8-GPU node under TP8; all recipes shard across 2 nodes (16x MI300X) with PP2xTP8 (~102 GB/GPU). gfx942 also requires VLLM_ROCM_USE_AITER_MLA=0. Recipes (each: run.sh + README.md + niah_probe.py): - pp2xtp8 PP2xTP8 baseline, no expert parallelism - wideep_int4_allgather +EP16, generic allgather_reducescatter all2all, a8w4 - wideep_int4_moriep +EP16, true MoRI-EP (mori_low_latency) all2all, a8w4 Uses the public vendor image amdsiloai/vllm:kimi-k3-mi325x-release-v2 (anonymous pull). Recipes only; no benchmark results included. Scripts are env-parameterized (IMAGE/MODEL_DIR/MASTER/PORT/...) and bash -n / py_compile clean. RDMA fabric env (NCCL_IB_HCA/GID, MORI_*) is cluster-specific and documented as overridable. AI-assisted (Claude); human-reviewed. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Ravi Gupta <ravi.gupta@amd.com>
…-node PP uses NCCL
…ays TP (not a product)
Adds scripts/vllm/kimik3_mi300x/wideep_disagg_2p2d/ — a prefill/decode DISAGGREGATED Kimi-K3 (MXFP4) recipe across 4 MI300X nodes: 2 prefill + 2 decode, TP2xDP8 -> EP16 per pool via MoRI-EP all2all (no pipeline parallelism), joined by the MoRIIO connector for prefill->decode transfer of both the MLA attention KV and the Kimi-Delta-Attention (KDA/mamba) recurrent+conv state. STATUS: WORK IN PROGRESS, lands as reviewable infra + honest STATUS.md, NOT a production deployment. The MoRIIO transport is byte-perfect (attention KV and KDA state arrive element-identical on all 8 decode ranks, verified by the probes under patchers/diagnostics/) and all connector-level bugs are fixed, but an open decode-side accuracy bug means exact long-context (NIAH) recall does not pass yet. Short single-token recall and coherent generation work; multi-token exact recall is wrong and non-deterministic at greedy temp=0 (non-determinism => forward-pass bug, below the sampler). STATUS.md carries the exact repro and the full ruled-out list; use the three colocated recipes for serving today. Connector fixes (folded into the vLLM source branch the image builds; also kept here as reference patchers under patchers/): - mamba block-id routing: transfer KDA state by the mamba KV-cache group's block ids, not the attention group's - remote_tp_size normalize: degenerate remote_tp_size<=1 -> world_size so KV fans out to all decode TP ranks - mamba N-1 boundary: producer computes h(N-1), decoder recomputes token N (mirrors vLLM's nixl/mooncake hybrid-PD handling) - gated diagnostics (default OFF): K3_WRITE_FENCE/K3_WRITE_DEVSYNC ordering, K3_MLA_SINGLE_SPLIT deterministic reduction (both inert on the known bug) Scripts are env-parameterized (node IPs/hostnames, MODEL_DIR, IMAGE); Dockerfile bases are ARG-driven placeholders. No benchmark results included. bash -n / py_compile clean; no local paths, IPs, tokens, or private tags. AI-assisted (Claude); human-reviewed. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Ravi Gupta <ravi.gupta@amd.com>
Bring the wideep_disagg_2p2d recipe from WIP to validated. Single-needle NIAH passes deterministically through 300K tokens (all depths) on the 2-prefill + 2-decode EP16 disagg serve. Two root-cause fixes for the decode-recall bug: - 4-KV-cache-group block routing (apply_kimik3_moriio_group_routing.py): K3 has 4 KV groups (3 KDA/mamba + 1 MLA); the connector hardcoded 2-group indices and sent MLA KV to mamba block-ids. Now routes each layer by its own group. - Multi-chunk prefill transfer (apply_kimik3_chunk_gate_fix.py + chunked_allgrp.py): the final-chunk gate used block count, which fires after chunk 1 when a prompt fits in <=1 padded block, so only max_num_batched_tokens of KV crossed. Now gates on compute progress from scheduler_output (map -> entry defer -> accum final-detect -> post-loop sweep). Also: run_2p2d.sh fabric now env-overridable (validated Thor2 bnxt defaults ON, THOR2_BNXT_FIX toggle) + LOAD_STRATEGY/chunk-gate knobs; STATUS.md rewritten to VALIDATED with the root-cause writeup; README winning-config + fabric-override + debugging sections; RESULTS.md (NIAH 10K-300K + latency/throughput 5.7x@8, 7.3x@16); benchmark_niah.py + concurrency_bench.py; niah_probe.py --ctx-list sweep. Remove 4 dead-end patchers (hma_fix, kda_fix, kda_offsets, sampler_fix). Co-Authored-By: Claude <noreply@anthropic.com>
Bump Dockerfile VLLM_REF to kimi-k3-wideep-disagg-fullsource-v2 — the base fullsource branch plus the two decode-recall fixes folded into vLLM source (4-KV-group block routing + multi-chunk compute-progress gate). The rebuilt image now has the fixes baked in; the runtime patchers stay in patchers/ and no-op idempotently on a v2 image (and still work on an older image). README updated to name the v2 branch and explain the baked-in vs patcher relationship. Co-Authored-By: Claude <noreply@anthropic.com>
…build crash)
The vLLM wheel build fails with `ValueError: invalid literal for int() with
base 10: ''` in setup.py compute_num_jobs when MAX_JOBS reaches the pip
subprocess empty (the ENV MAX_JOBS=${MAX_JOBS} did not propagate under the
legacy builder). Set MAX_JOBS/NVCC_THREADS inline on the pip install command so
the build is reproducible regardless of builder.
Co-Authored-By: Claude <noreply@anthropic.com>
…timization + scratch-build validation - RESULTS.md: single-needle NIAH now recorded to 500K (3/3 PASS ~301s); 900K documented as a known hang (prefill freezes, GPUs 100%, no result; serve recovers on client kill — confined to the one oversized request). Wall is between 500K and 900K. - OPTIMIZATION.md (new): validated KV-cache lever — KV_CACHE_MEMORY_BYTES 8e9->40e9 raises the GPU KV cache 542K->2.84M tokens (5.2x, 2.84x concurrency at 1M ctx), required for single requests >~600K; the 8e9 pin was only a profile_run-hang workaround, not a mem limit. Plus max_num_seqs guidance, the batched-tokens dead end, the single-stream-latency trade, the residual write race, and base-image digest-pin durability. - README: add KV_CACHE_MEMORY_BYTES to the config table + link OPTIMIZATION.md; note MAX_MODEL_LEN=1000000 for full native ctx. - STATUS: note the recipe was validated from a from-scratch image build (fixes baked into the v2 vLLM branch; patchers idempotent no-ops on it). Co-Authored-By: Claude <noreply@anthropic.com>
…AH to 900K) Contexts above ~500K used to hang. py-spy root-caused it to gather_initial_states running a bool((indices>=n).any()) device->CPU sync per KDA layer per prefill chunk (~25k full stream drains at 750K) purely to log a warning; the index clamp above already made the GPU address safe. Gate the diagnostic behind K3_KDA_GATHER_LOG=1 (default OFF); correctness unchanged. Result: single-needle NIAH now PASSES the full native range 10K-900K (500K 301s, 750K 542s, 900K 717s; sub-quadratic), where 750K/900K previously hung indefinitely. - Fold: vLLM branch kimi-k3-wideep-disagg-fullsource-v3 (Dockerfile VLLM_REF bumped) - Ship: patchers/apply_kimik3_kda_gather_nosync.py + wired into run_2p2d.sh (no-op idempotent on a v3 image) - Docs: STATUS/RESULTS/OPTIMIZATION updated to the 10K-900K range + the fix writeup; noted batched=8192 still corrupts generation (keep 2048). Co-Authored-By: Claude <noreply@anthropic.com>
…kage) - concurrency_bench.py: hardcoded router IP -> ROUTER_URL env var (default 127.0.0.1), matching the other test scripts. - run_2p2d.sh: drop internal cluster name from a fabric-defaults comment. - kimi_k3_moriep_report.html: replace node hostnames, an internal home path, and the internal cluster name with generic descriptions. No credentials were ever committed; this removes the remaining environment-specific identifiers so the recipe reads as a generic MI300X + Thor2 reference. Co-Authored-By: Claude <noreply@anthropic.com>
Remove runtime patchers, HTML report, and redundant STATUS/OPTIMIZATION docs. Connector fixes are image-only (VLLM_REF=v3). Default OCI mlx5 fabric env. Add Kimi-K3-MXFP4 to scripts/vllm_dissag/models.yaml and cross-link MI300X recipes from benchmark/kimi_k3/README.md.
MoRI wheel build failed with AssertionError: run() must precede build_extension() when pip used an isolated build env without torch.
… lean image Wire Kimi-K3 MXFP4 disaggregated serving (MI300X/gfx942) into MAD's scripts/vllm_dissag/ two-axis driver, and add the lean-image build path so the recipe carries config only (fixes baked into the image, not 33 runtime patchers). vllm_dissag integration: - moriio.sh: K3 TP2xDP8-per-pool topology, moriio_pod_hosts in KV config, kv-transfer-config on headless workers, router --moriio-dp-size, safe model_args tokenizer (_model_config_to_array via shlex) replacing bare eval that mangled --quantization-config JSON. - models.yaml: Kimi-K3-MXFP4 recipe entry (TP2, DECODE_CUDAGRAPH_MODE=NONE). - vllm_disagg.sh / run_xPyD_models.slurm: TP2xDP8 rank mapping, K3 in VALID_MODELS, 2P/2D + MoRI-EP validation, prefill/decode JIT cache split (shared AITER .so names). - run_interactive.sh / drive_cell.sh: JIT cache split by role, DECODE_CUDAGRAPH_MODE passthrough, RDMA mount fix. - rixl.sh: adopt shared model_args tokenizer. - argv_assert.sh: K3 dry-run assertions (TP2, DP8, pod hosts, api-server-count=dp_size, reasoning parser); 37 passed / 0 failed offline. - ARCHITECTURE.md / README.MD: K3 worker taxonomy (W1-W5), 4-node layout, build/run docs. Lean image (Session 3): - build_lean.sh: WITH_NIXL=0, VLLM_REF=kimi-k3-wideep-disagg-fullsource-v3 -> kimik3-wideep-disagg-lean:latest (MoRI-EP only). - SESSION3_LEAN.md: lean deliverable + baked-in fixes vs 33 patchers. - run_2p2d.sh / run_2p2d_launch.sh: drop runtime patcher deployment, per-node MODEL_DIR override, OCI mlx5 fabric defaults. Validated: argv_assert 37/0-failed offline; NIAH 9/9 @ 10K/50K/100K on job 223124 (F18). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lean, upstream-ready version of the Kimi-K3 MI300X (gfx942) serving recipes — the config-only successor to #193. Same recipes (3 colocated + 2P/2D EP16 MoRIIO disagg, NIAH to 900K), but all vLLM/MoRI source fixes are baked into the image, so the tree carries zero runtime patchers (replaces #193's 33
apply_*.pypatchers). Also integrates the disagg path into MAD'sscripts/vllm_dissag/launcher framework (GLM #206 pattern).Central design rule: the recipe is config only (
models.yaml+run_2p2d*.sh/vllm_dissag); the image (kimik3-wideep-disagg-lean:latest,VLLM_REF=kimi-k3-wideep-disagg-fullsource-v3,WITH_NIXL=0) carries the fixes.What's in this PR (33 files, +2289/−26)
scripts/vllm/kimik3_mi300x/colocated (pp2xtp8,wideep_int4_allgather,wideep_int4_moriep) + disagg (wideep_disagg_2p2d) with README/RESULTS,build_lean.sh,SESSION3_LEAN.md. No patchers in tree.scripts/vllm_dissag/Kimi entry inmodels.yaml, MoRIIO K3 TP2×DP8 topology (connectors/moriio.sh), quant-config JSON tokenization fix, JIT prefill/decode cache split,DECODE_CUDAGRAPH_MODE=NONEfor decode, offlinetests/argv_assert.sh.build_lean.sh; Dockerfile builds the fullsource-v3 vLLM branch. Binary produced out-of-band and pushed to operator registry (no duplicated layers in-repo).Baked-in fixes (no runtime patchers)
kimi-k3-wideep-disagg-fullsource-v3(raviguptaamd/vllm) folds in: 4-KV-group block routing, multi-chunk prefill transfer, KDA gather sync-free (unblocks >500K), mamba block-id / remote_tp / N−1 connector fixes. PR #193's patchers no-op against this image.Validation (MI300X, OCI)
argv_assert.shgreen; cluster-residue scrubbed (no hardcoded IP/host/user paths).Companion PR
vLLM sidecar:
raviguptaamd/vllmbranchkimi-k3-wideep-disagg-fullsource-v3(the source fixes this image builds).Relationship to #193
Supersedes the investigation-stage #193 with a config-only, patcher-free tree suitable for
develop. Opened as draft pending internal review.Test plan
build_lean.shon a gfx942 build nodevllm_dissaglive smoke (Kimi entry)