Repository navigation
Matmul: the routed product is tuned by a tuner of its own - #1018
Merged
Merged
Conversation
SamuelBelanger
marked this pull request as ready for review
October 29, 2026 19:44
SamuelBelanger
force-pushed
the
refactor/routed-own-tuner
branch
from
November 5, 2026 15:21
fcb2929 to
6f30afa
Compare
louisfd
approved these changes
Nov 5, 2026
A routed product rode the dense matmul call as a flag (MatmulCall.routes), keyed as a dense product of batches of one row, so every dense candidate guarded against it. It now has its own key, candidate list, floors per stack kind and tuner (routed::launch_tuned), the way the m-grouped product has. The dense call drops routes, and MatmulBatching::Slabs leaves the dense key. Candidates are measured with routes spread evenly over the experts, uploaded rather than launched, so they stay valid in a build that discards launches. A round stops near the roofline of the rows, the output and the slabs the routes can reach. A product no floor serves is refused before racing. Routed tune records move to their own tuner, so routed keys re-tune once.
SamuelBelanger
force-pushed
the
refactor/routed-own-tuner
branch
from
November 5, 2026 16:01
6f30afa to
976bc25
Compare
This was referenced Nov 5, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1007; merge after it.
A routed product rode the dense matmul call and tables as a flag (
MatmulCall.routes), keyed as a dense product of batches of one row, so every dense candidate guarded against it. It now has its own key (rows, slots, experts, the stack's kind and layout), candidate list (plane shuffle, cube staged, unit registers, float and packed), floors per stack kind, and tuner (routed::launch_tuned), the way the m-grouped product has. The dense call dropsroutes, andMatmulBatching::Slabsleaves the dense key.Ceiling::Either(before, it was the ceiling of whichever dense autotune served the call; the few-rows one bounds by memory alone).Listingand its list rule. Both keys anchor their rows where they are built, so a bucket does not move with the autotune level.Every candidate that ran routed before is still raced; none is lost. Routed tune records move to a tuner of their own, so routed keys re-tune once.
Main's autotunes replaced the tune tables, so the routed arms are removed from their float, packed and block-scaled candidates and the roofline.
Tested on an RTX 2060 (CUDA) after merging #1007's head: the indexed, routed, m-grouped, packed and autotune tests, including tuned routed products on float, q8, q4, nvfp4 and mxfp4 stacks against the host reference, the spread-routes generator, and the full cubek-matmul suite on the CPU runtime, where the tuned products read real packed stacks too. Clippy clean.
autotunes.txtunchanged. On that card a pinned routed plane shuffle declines an MXFP4 stack at the test's shapes (it needs unit columns that close n against a packed rhs); that is a decline in unchanged code, and the tuned product is served by another candidate.