Skip to content

perf(py-lsp): O(1) type membership in the shared Python cross-registry build - #2420

Open
Fieldnote-Echo wants to merge 4 commits into
DeusData:mainfrom
Fieldnote-Echo:perf/py-cross-registry-index
Open

Fieldnote-Echo wants to merge 4 commits into
DeusData:mainfrom
Fieldnote-Echo:perf/py-cross-registry-index

Conversation

@Fieldnote-Echo

@Fieldnote-Echo Fieldnote-Echo commented Sep 28, 2026 •

Copy link
Copy Markdown

Fixes #2413

What does this PR do?

The shared Python cross-registry build registers defs one at a time, so each Method's receiver probe (internal/cbm/lsp/py_lsp.c:5049) is a linear scan over the types registered so far (type_registry.c:830-834). I back only that existence probe with a cbm_ht set of type QNs, updated after every cbm_registry_add_type, as the Go (959f4de) and Rust (10b3b79) fixes did. If the set cannot be allocated or an insert fails, probes fall back to the registry lookup. Keeping the per-def order makes the registry identical by construction (a types-first build would be simpler but can reorder auto-registered receiver types). Sorted node and edge dumps match main byte for byte, and peak RSS is unchanged.

Linux x86_64, 32 logical CPUs, GCC 16 main this PR
Python builder, Python 3.14 stdlib + site-packages, 3 runs 67-75 s 0.20-0.25 s
Same builder, generated 2k / 8k / 32k classes 0.11 / 1.5 / 22 s 8 / 28 / 135 ms
Whole index wall time, Python 3.14 tree 175-182 s 113 s

pylsp_scale_cross_registry_build_linear fails on main: going from 1k to 16k classes costs over 100× more time there, and under 8× here. It is in the py_lsp_scale perf suite, which PR CI skips via CBM_SKIP_PERF, so it needs a local run. pylsp_cross_registry_interleaved_order pins the registry order.

Checklist

  • Every commit is signed off (git commit -s) — required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test)
    (run as make -f Makefile.cbm test-par)
  • Lint passes (make -f Makefile.cbm lint-ci)
    (for this PR's changes: CI's lint job passes on this head, and local cppcheck 2.22 / clang-format 23 add no findings over main)
  • New behavior is covered by a test (reproduce-first for bug fixes)

Prepared with AI assistance (exploration, reproduction, verification, implementation); the spec and design decisions are mine, and I reviewed and ran everything above.

…y build

The shared tier-2 registry registered Python defs one at a time without a
mid-build finalize, so every Method's receiver probe scanned all registered
types linearly: O(methods x types), single-threaded in lsp_cross_prepare.
Keep the per-def registration order and back only the existence probe with a
cbm_ht set of type QNs, as the Go and Rust builders do; the registry is
identical by construction.

On the Python 3.14 stdlib + site-packages the builder drops from 67-75 s to
0.20-0.25 s, with a byte-identical graph.

Signed-off-by: Nelson Spence <nelson@projectnavi.ai>
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

Signed-off-by: Nelson Spence <nelson@projectnavi.ai>
Signed-off-by: Nelson Spence <nelson@projectnavi.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(py-lsp): shared Python cross-registry build is O(methods × types)

1 participant