Skip to content

Matmul: the routed product is tuned by a tuner of its own - #1018

Merged
SamuelBelanger merged 1 commit into
mainfrom
refactor/routed-own-tuner
Nov 5, 2026
Merged

SamuelBelanger merged 1 commit into
mainfrom
refactor/routed-own-tuner

Conversation

@SamuelBelanger

@SamuelBelanger SamuelBelanger commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1007; merge after it.

A routed product rode the dense matmul call and tables as a flag (MatmulCall.routes), keyed as a dense product of batches of one row, so every dense candidate guarded against it. It now has its own key (rows, slots, experts, the stack's kind and layout), candidate list (plane shuffle, cube staged, unit registers, float and packed), floors per stack kind, and tuner (routed::launch_tuned), the way the m-grouped product has. The dense call drops routes, and MatmulBatching::Slabs leaves the dense key.

  • Candidates are measured on the call's lhs and stack, at the call's own rows, with routes spread evenly over the experts and uploaded, not launched: in a build that discards launches the routes are valid indices (the lhs and stack still hold whatever their buffers held). Spread routes read the most slabs, so the measurement and the bound are the bandwidth-heaviest case; a call whose routing is skewed reads fewer slabs and could prefer another winner. That is a known limit of a measured-on input that cannot trust the call's routes.
  • A round stops near the roofline of the rows, the output and the slabs the routes can reach, always against Ceiling::Either (before, it was the ceiling of whichever dense autotune served the call; the few-rows one bounds by memory alone).
  • A product no floor serves is refused before racing, checking floors until the first serves. The unit registers refuse a packed stack, so a device neither other routine prepares on has no floor for one. A caller falls back as it does on any refusal. The routed floor check still allocates the output it prepares against.
  • The m-grouped and routed listings share one Listing and its list rule. Both keys anchor their rows where they are built, so a bucket does not move with the autotune level.

Every candidate that ran routed before is still raced; none is lost. Routed tune records move to a tuner of their own, so routed keys re-tune once.

Main's autotunes replaced the tune tables, so the routed arms are removed from their float, packed and block-scaled candidates and the roofline.

Tested on an RTX 2060 (CUDA) after merging #1007's head: the indexed, routed, m-grouped, packed and autotune tests, including tuned routed products on float, q8, q4, nvfp4 and mxfp4 stacks against the host reference, the spread-routes generator, and the full cubek-matmul suite on the CPU runtime, where the tuned products read real packed stacks too. Clippy clean. autotunes.txt unchanged. On that card a pinned routed plane shuffle declines an MXFP4 stack at the test's shapes (it needs unit columns that close n against a packed rhs); that is a decline in unchanged code, and the tuned product is served by another candidate.

@SamuelBelanger
SamuelBelanger marked this pull request as ready for review October 29, 2026 19:44
@SamuelBelanger
SamuelBelanger force-pushed the refactor/routed-own-tuner branch from fcb2929 to 6f30afa Compare November 5, 2026 15:21
Base automatically changed from fix/m-grouped-packed-cube-staged to main November 5, 2026 16:01
A routed product rode the dense matmul call as a flag (MatmulCall.routes), keyed as a dense product
of batches of one row, so every dense candidate guarded against it. It now has its own key,
candidate list, floors per stack kind and tuner (routed::launch_tuned), the way the m-grouped
product has. The dense call drops routes, and MatmulBatching::Slabs leaves the dense key.

Candidates are measured with routes spread evenly over the experts, uploaded rather than launched,
so they stay valid in a build that discards launches. A round stops near the roofline of the rows,
the output and the slabs the routes can reach. A product no floor serves is refused before racing.

Routed tune records move to their own tuner, so routed keys re-tune once.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants