Repository navigation
Conversation
DeusData
added
bug
DeusData
added
parsing/quality
Owner
|
Thanks for this — there is clearly a lot of substantial work here, and I want to be straight with you about the review problem it creates rather than let it sit silently. As it stands the PR spans 17 top-level directories: The title promises incremental-indexing parity and a 3x edit-indexing win, and that is a change I am genuinely interested in. But it is currently bundled with a benchmark harness, documentation, packaging and tooling changes, and this project reviews atomically — one PR, one claim. A change of this size touching this many subsystems cannot be reviewed responsibly as a unit, and the incremental-indexing work deserves better than being approved or rejected on the strength of everything around it. Could you split it? The natural seam looks like: (1) the incremental-indexing correctness and performance change in Two mechanical notes: the branch is CONFLICTING against I realise splitting completed work is annoying, and I would rather ask than quietly let a 124k-line PR go stale. The incremental-indexing piece is the part I most want to see land. |
Restore incremental_aborts_when_previous_coverage_is_unreadable in tests/test_pipeline.c. The test renames index_coverage.detail, appends MustNotBeIndexed to helper.go, and requires cbm_pipeline_run to fail while the committed node count remains unchanged. Use incremental_test_config, cbm_strdup, CBM_PATH_MAX, and cbm_fopen so the upstream assertion follows the merged configuration and portable filesystem paths. Verification: focused ASan/UBSan run passed 1 test with 390 filtered; complete pipeline suite passed all 391 tests; git diff --check passed against be89d49 and 97ce23f. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The watcher stored one repository-wide dirty signature in each file row while leaving observed_mtime_ns and observed_size at zero. Incremental fallback rows recorded mtime and size but left observed_hash empty. Four translation units also carried separate Darwin, Windows, and POSIX struct stat field branches, and the pipeline hash reader used fopen instead of the UTF-8-safe wrapper. Move the canonical XXH3-64 file reader to src/foundation/compat_fs.c, open paths with cbm_fopen, and centralize nanosecond conversion in cbm_stat_mtime_ns. Use those helpers from pipeline, watcher, git snapshot, and MCP coverage freshness. Changed or untracked files now record the same content hash, mtime, and size contract; missing/deleted files retain empty/zero observations. Strengthen incremental_publish_failure_keeps_existing_db and watcher_marks_dirty_file_before_failed_index_callback to compare persisted hashes and stat metadata with the source files. Verified: ASan/UBSan pipeline 391/391; watcher 83/83; focused MCP coverage freshness 1/1; make test-foundation exit 0; lint-source-safety exit 0; security-audit.sh exit 0. Scoped MinGW syntax over all 10 changed translation units exits 0; -Werror additionally encounters the inherited src/foundation/compat_fs.c:1202 -Wtype-limits warning. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add cbm_git_run_argv and cbm_git_resolve_executable in src/git/git_command.c. The runner constructs git --no-optional-locks -C <repo> argv, supervises the owned cbm_subprocess_t through terminal containment, and captures requested stdout in an owner-cleaned temporary file. Migrate src/git/git_context.c, src/git/git_snapshot.c, and src/mcp/mcp.c from interpolated shell command strings. detect_changes invokes diff and status directly, rejects leading-dash revision names, terminates revision parsing with --, and deduplicates paths in C. Reuse the existing hardened Windows git.exe resolver from src/watcher/watcher.c without changing watcher process ownership. Strengthen tests with real repositories whose paths contain spaces and %!^&; and a Git-valid branch containing %!&;. Assert exact Git operation counts, temporary-file cleanup, literal metacharacter handling, and long-path watcher behavior. Verified: make -f Makefile.cbm test exited 0; MCP 287/287; watcher 83/83; git_context 5/5; schema-declared keys 3/3; daemon_runtime 43/43; disconnect cancellation 20/20 before and after; changed production translation units passed MinGW -Wall -Wextra -Werror -fsyntax-only. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Capture the canonical source checkout in tf_capture_repository_root() before any suite changes process CWD. The bounded ancestor walk requires both Makefile.cbm and vendored/xxhash/xxhash.h, and handles POSIX and Windows path separators. Make clsp_nocrash_issue355_xxhash_header build an absolute CBM_PATH_MAX-bounded fixture path and open it through cbm_fopen(). This removes its run-from-repository-root assumption without changing production LSP execution. Verified tests/test_main.c and tests/test_c_lsp.c with warning-clean ASan/UBSan compilation, the exact issue355 test (1/1) and complete c_lsp suite (752/752) from /private/tmp, optional MinGW -Wall -Wextra -Werror syntax checks, and the complete ASan/UBSan suite (exit 0). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace CLOCK_MONOTONIC wall time in tests/test_py_lsp_scale.c with the ISO C clock() process-time interface. Scheduler pauses during the 2,000-class sample can no longer fabricate a quadratic resolver regression. Reject unavailable or backward clock samples before evaluating the existing 100/500/2,000-class ratio. Fixture sizes, resolution assertions, and the ratio < 100 quadratic guard remain unchanged. Verified with warning-clean ASan/UBSan compilation, five consecutive focused runs (ratios 34.3x-40.0x), optional MinGW -Wall -Wextra -Werror syntax checking, and the complete ASan/UBSan suite (exit 0). Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Route cbm_safe_getenv() and cbm_getenv_fits() through one cbm_platform_read_environment_value() classifier in src/foundation/platform.c. POSIX performs one allocation-free environ scan; Windows keeps GetEnvironmentVariableW plus UTF-8 conversion, and both public APIs distinguish missing, empty, present, and too-long values without silently falling back to a different configuration directory. Add tests/test_platform.c coverage for present, empty, missing, exact-fit, overlong, and Windows wide-character values. Verified with a warning-clean ASan/UBSan build, platform 23/23, the overlong CLAUDE_CONFIG_DIR CLI regression, the complete ASan/UBSan suite, optional MinGW syntax checking, and clean diffs against both merge parents. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace cbm_mcp_auto_index_within_limit()'s private cbm_discover_count_bounded() policy with cbm_mcp_auto_index_within_configured_limit(), the same five-second bounded admission helper used by src/daemon/application.c. Keep MCP-specific block state and structured warnings local while preserving auto_index_limit=0 as unlimited. Report the first rejected cardinality in build_project_list_error_srv(): auto_index_limit=1 now says 'at least 2 indexable files' instead of displaying the saturated limit. tests/test_mcp.c pins both the observed cardinality and configured value. Verified with a fresh warning-clean ASan/UBSan build, MCP 287/287, focused daemon configured-limit admission, focused explicit-path admission, git diff --check, and the historical intent in commit 6daeb7b. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Define TEST_INCLUDE_FLAGS as the existing -Itests -Itests/repro pair and use it in the ASan/UBSan, nosan, TSan, and Clang analyzer test compilation recipes. tests/test_index_resilience.c can now resolve tests/repro/repro_harness.h under test-analyze without duplicating include policy. The complete make -f Makefile.cbm test-analyze run finished with no compiler or fatal errors. Existing Clang checker warnings remain visible for separate triage; the change affects only compiler header lookup and has no runtime or platform-specific behavior. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Write the complete test-analyze compiler stream to build/c/analyze-report.txt, save the compiler status before filtering diagnostics, and exit with that status. A fatal compiler error can no longer be converted into success by grep, while warning/error/note output and the quiet 'No issues found.' result remain visible. Use portable POSIX shell status capture instead of PIPESTATUS. Verified that CC=false and a missing compiler return nonzero, CC=true returns zero, and the complete real Clang analyzer finishes without compiler or fatal errors. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a test-syntax target that requires TEST_SYNTAX_SRCS and invokes the selected C translation units with CFLAGS_NOSAN plus TEST_INCLUDE_FLAGS. Ad hoc syntax checks now inherit the project's warning policy, platform flags, include roots, and six gated test-API definitions instead of reconstructing them by hand. Verified that an empty source list fails, tests/test_mcp.c plus src/mcp/mcp.c pass with native Clang, clearing EDITOR_TEST_DEFINES exposes the expected cbm_hook_path_contains_for_testing errors, and src/mcp/mcp.c passes with x86_64-w64-mingw32-gcc. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Move tests/test_mcp.c's extern char **environ declaration back into the non-_WIN32 include branch, matching parent 97ce23f. The merge had displaced it near the supervisor test helpers, where MinGW reports a dllimport attribute mismatch under -Werror. Verified with Makefile.cbm test-syntax for tests/test_mcp.c under native Clang and x86_64-w64-mingw32-gcc. The focused index_supervisor_start_failure_is_fail_closed_in_real_host test passes on the fresh ASan/UBSan runner. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace Makefile.cbm's four Bash-only PIPESTATUS recipes with one run_logged_command macro. The command side records its exit code in a per-process sidecar while tee streams the report; the recipe returns the command failure first, otherwise tee's failure, and removes the sidecar through a POSIX trap. Add tests/test_makefile_logged_command.sh to scripts/test.sh's fast contracts. It proves success logging, exit-7 propagation, report-write failure, and cleanup under /bin/sh and /bin/dash. A focused macOS test-leak TEST_LEAK_SUITES=arena run passed 31 tests and reported 0 leaks. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add cbm_json_escaped_len beside cbm_json_escape in src/foundation/str_util.c and replace the private helpers in src/git/git_context.c, src/pipeline/pass_definitions.c, and src/pipeline/pass_parallel.c. Both pipeline appenders now call the shared whole-string writer, so atomic buffer admission and emitted byte counts use one control-character policy instead of duplicated 1-byte substitutions versus 6-byte \u00XX escapes. tests/test_str_util.c compares length and writer output for every non-NUL byte. tests/test_pipeline.c proves form-feed JSON round-trips through git context. Fresh ASan/UBSan focused suites passed, oversized definition properties passed in forced sequential and parallel routes, and all changed translation units passed native Clang and MinGW -Werror syntax checks. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: src/mcp/mcp.c privately defined query_max_output_bytes and 32768 while src/cli/cli.c repeated both literals in CBM_CONFIG_REGISTRY, allowing handler and advertised defaults to drift. Define CBM_CONFIG_QUERY_MAX_OUTPUT_BYTES, CBM_DEFAULT_QUERY_MAX_OUTPUT_BYTES, and its string twin in src/cli/cli.h. Consume them from the registry and MCP handler, and remove the misleading literal 100000 from query_max_rows guidance without changing the Cypher engine policy. tests/test_cli.c adds cli_config_registry_query_limits_use_shared_definitions, which binds the registry key/default to the shared definitions and checks the non-bypassable-ceiling guidance. Verification: 323 CLI and agent_clients tests passed; Makefile.cbm test-syntax passed for tests/test_cli.c, src/cli/cli.c, and src/mcp/mcp.c; lint-source-safety passed; git diff --cached --check passed. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/cypher/cypher.c:5927 replaces the fixed 100000-row execution ceiling with cbm_cypher_limits_t. Final max_rows shaping now returns a complete prefix with truncated=true, while intermediate scan, expansion, and UNION exhaustion returns 'query exceeded the working-row budget' instead of partial matches. src/store/store.c:2844 adds a cached ORDER BY id LIMIT statement and cbm_store_close finalizes it. Variable-length traversal requests remaining rows plus one sentinel instead of the inherited 100-row BFS prefix. UNION branches retain the working budget until concatenation and DISTINCT finish, then apply the single final output cap. src/foundation/constants.h:93 and src/cli/cli.c:14184 define and validate query_max_rows and query_max_working_rows through the existing CBM config registry. src/mcp/mcp.c:8015 preserves the query_graph input schema, emits structured truncation metadata, and returns MCP tool execution errors for incomplete intermediate work. The boundary follows the MCP 2025-11-25 tools specification's isError execution-error contract, Neo4j's transaction-memory termination guidance, and Cypher ORDER BY/LIMIT semantics. No platform-specific APIs are added; bounds use portable C integers and SQLite prepared statements. Tests: ASan/UBSan test-runner build; Cypher 194/194; query_graph MCP 17/17; tool consolidation 114/114; CLI row-limit validation 1/1; installed skill budget contract 1/1; source syntax, source safety, and git diff --check. The macOS leaks task-port failure remains an explicitly inconclusive environment gate for the later full checkpoint. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/depindex/depindex.c:637 replaces rank_by_import_usage's CBM_DEP_PROJECT_IMPORT_FETCH_LIMIT=500 search with cbm_store_visit_nodes_by_label. Commit fff8ea2 introduced the capped search while promising most-imported package selection; imports sorted beyond that window could not affect the result. count_candidate_import maps only discovered candidate names to counters inside the ranking array. The store streams all project Variable rows, then qsort retains the existing import-count/name ordering. Runtime is O(V + C log C) and auxiliary memory is O(C), where V is project Variable rows and C is discovered candidates; no OS-specific API or unbounded full-node array is added. tests/test_depindex.c:1331 places 501 pkg-z imports after 500 pkg-a imports. The committed implementation failed RED with 'pkg-a' != 'pkg-z'; the streaming implementation ranks pkg-z first. ASan/UBSan build, focused regression 1/1, full depindex 43/43, source safety, and git diff --check pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/depindex/depindex.c:1058 removes the CBM_DEP_LINK_IMPORT_FETCH_LIMIT=500 search introduced by fff8ea2. cbm_dep_link_cross_edges now streams every project Variable node and preserves per-edge file ownership instead of silently omitting imports after the first 500 search rows. src/store/store.c:2871 factors the existing label visitor through one five-column SELECT and adds cbm_store_visit_node_refs_by_label for ID-bearing callbacks. The existing cbm_store_visit_nodes_by_label signature and behavior remain intact; both paths finalize the SQLite statement on callback or step failure. Import linking remains O(I) after the existing Module name hash is built and uses O(1) additional import memory rather than an array of full node rows. The callback uses portable C11 types and the existing SQLite/store APIs; no platform-specific branch is added. tests/test_depindex.c:1298 creates 501 imports and proved RED at 500 linked edges. The streaming path creates 501 IMPORTS edges and 501 file-owner rows. ASan/UBSan build, focused regression 1/1, store_nodes 121/121, depindex 44/44, selected syntax, source safety, and git diff --check pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
src/depindex/depindex.c:1018 replaces the CBM_DEP_LINK_MODULE_FETCH=100000 bulk search introduced by ce2ba50 with a single ranked Module visitor. collect_dep_module folds duplicate names to highest PageRank then lowest node ID, and cbm_dep_link_cross_edges frees every owned hash entry on success and visitor failure. src/store/store.c:2871 and src/store/store.h:588 expose ID, borrowed identity fields, and PageRank through cbm_store_visit_ranked_node_refs_by_project_pattern_and_label. The SQLite statement is finalized on prepare, callback, step, and success paths without materializing cbm_node_t rows or adding platform-specific code. The linker remains expected O(M + I) time and O(U) auxiliary memory for M dependency Modules, I project imports, and U unique Module names. It removes the previous correctness ceiling and avoids the O(M log M) global sort that a relevance-ordered replacement would require. tests/test_store_nodes.c:760 checks project-pattern and label filtering plus rank delivery. tests/test_depindex.c:1154 checks highest-PageRank and lowest-ID duplicate selection, edge target identity, and file ownership. ASan/UBSan store_nodes 122/122 and depindex 44/44 pass; selected -Werror syntax, scripts/check-source-safety.sh, and git diff --check pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The 100-class CPU-time baseline varied enough to make the 35x scale guard fail intermittently even when the 500-to-2000 interval remained well below quadratic growth. Retain the 35x 100-to-2000 guard and confirm any violation against an 8x bound over 500-to-2000. Clear quadratic growth is 16x over that interval, so the detector keeps a 2x failure margin while rejecting small-baseline noise instead of weakening to upstream main’s 200x allowance. Verified with a reproduced pre-fix failure, 12 consecutive production runs, a focused ASan/UBSan run, and the complete CI lint gate. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
DeusData
added a commit
that referenced
this pull request
Aug 21, 2026
Attacked the inputs rather than the patterns this time. Three worked.
ONE INVALID BYTE HID A WHOLE FILE. The scanner abandoned any file that
failed to decode as UTF-8, so appending a single 0xFF made it skip every
readable line in that file, plaintext payload included. A complete
evasion costing one byte. Files are now decoded with replacement rather
than abandoned, and a NUL byte -- git's own binary heuristic -- is what
marks a file as genuinely not a review surface. Only three tracked files
reach that path today: a PNG, a Windows ETW manifest and the nomic blob.
Better still, the evasion is now its own signal: a file with a TEXT
extension that is not valid UTF-8 is reported, because a stray byte in a
.md or .c is anomalous regardless of what surrounds it.
THE TRIPWIRE MATCHED CASE-SENSITIVELY. `Scripts/evil.sh` and
`.GitHub/workflows/` walked past it, and on a case-insensitive checkout
those are the same files as the guarded ones. Now matched with POSIX
character classes rather than `${v,,}` (bash 4 only) or `tr` (external),
so the logic can be tested on any shell -- which matters for a gate
nobody can run locally the way CI runs it.
THE FILES ENDPOINT CAPS AT 3000. A pull request padded past that limit
would hide a CI change in the tail, and the gate would report green over
a change set it never saw. It now compares what the API returned against
the count the PR itself declares and REFUSES when they disagree, rather
than passing on partial data.
All three are pinned in the selftest, alongside the requirement that a
PNG with invalid UTF-8 stays silent -- the hardening must not turn every
binary into a finding.
Verified end to end against real pull requests: #1422 refused (4 guarded
paths), #1245 refused (42 guarded, 392/392 received so no truncation),
#1778 refused (2 guarded), #1678 passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Previous behavior: this branch lacked upstream SQL/dbt relation extraction, Pi tool metadata, monorepo watcher fixes, Swift extraction, startup contention recovery, and packaging updates. Taking upstream wholesale would regress the branch's configurable clustering, dynamic writers, MCP catalog, fail-closed watcher ledger, and bounded response behavior. Integrate Table/View/Model and dbt lineage through the canonical registry, LSP invalidation, sequential/parallel resolution, store, and architecture paths. Preserve the config-aware MCP catalog and trace_call_path compatibility while adding registry-derived Pi schemas and abort propagation. Combine monorepo-aware watcher paths with the dynamic dirty ledger, and fail loudly when UI response buffers saturate. The result is a tested superset of both exact parents without fixed preprocessing or graph cutoffs. Focused ASan/UBSan verification covers SQL/dbt lineage, per-file resolver caching, watcher paths, architecture boundaries, legacy CLI/MCP compatibility, source safety, and bounded HTTP failures; the pre-final broad candidate passed 8,782 tests with 4 skips. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior: the merged SQL relation predicates, lineage resolver call, and SQL language initializer did not match the pinned clang-format layout, so the PR lint job exited with status 2. Reflow only those expressions and adjacent string-literal boundaries without changing the resulting SQL, initializer values, or resolver arguments. Files: internal/cbm/lang_specs.c formats the SQL initializer; src/pipeline/registry.c aligns the lineage resolver call; src/store/store.c formats relation predicates and architecture SQL. Verification: lint-format, lint-no-suppress, source-safety, cppcheck, focused syntax, and 1,162 ASan/UBSan architecture/registry/language/extraction/pipeline tests pass. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The pipeline test sink appended worker logs to one global buffer without synchronization. Parallel extraction could therefore race strlen against snprintf, which both Linux TSan jobs reported. Guard each append with the existing portable cbm_mutex_t and bind its lifecycle to each capture window. Production logging, captured messages, benchmark inputs, and algorithm behavior are unchanged. Verification: - 10 focused four-worker TSan repetitions - make -f Makefile.cbm test-tsan - make -f Makefile.cbm test - make -f Makefile.cbm test-leak (0 bytes leaked) - make -f Makefile.cbm lint-ci test-compositional security - make -f Makefile.cbm test-analyze lint-source-safety Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Previous behavior could stop an ephemeral daemon when its final application session closed even though version-matched clients were already accepted or still coordinating before HELLO admission. Short outer hook deadlines could also hide the established fail-open warning before daemon connection. Add bounded cohort activity leases, a tri-state runtime stop guard, and host drain integration. Reuse the client cohort manager for activity and lifetime leases, bound bootstrap probes by the remaining deadline, and start hook execution timing only after authenticated daemon connection. Preserve existing legacy warnings and fail-loud endpoint diagnostics. Cover final-session, accepted-peer, provisional-session, bootstrap-deadline, cold CLI/MCP storm, and startup-diagnostic behavior. The added work and storage are fixed per process or session, so asymptotic runtime and memory bounds remain unchanged while duplicate manager allocation is removed. Verification: make -f Makefile.cbm BUILD_DIR=build/c-coldstorm test (8789 passed, 4 skipped); focused TSan (157 passed); lint-format; lint-source-safety; lint-cppcheck; security; parent-neutral 21-run benchmark output parity. Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
This was referenced Sep 20, 2026
pull Bot
pushed a commit
to TKaxv-7S/codebase-memory-mcp
that referenced
this pull request
Sep 23, 2026
… qualifiers Distilled from DeusData#1245 by Andrew Hundt (c51e5cb, the extract_defs.c half; the mcp.c and token-reduction parts are not taken). For C-family functions return_type was the text of the declaration's `type` node alone. In these grammars the `*`, `&` and `&&` live on the declarator and cv-qualifiers are sibling nodes, so const char *get_name(void) published char robj **lookupKeys(...) published robj Text &operator=(const Text &) published Text and no C or C++ function in any index had a pointer return type at all. The C/C++ resolver's return-type parser already strips a leading const/volatile and handles trailing `*`, `&`, `&&` -- branches that were unreachable for functions, so every `Foo *f()` was registered as returning Foo by value. c_declared_return_type() renders the declared type at both sites (functions and methods) for C, C++, CUDA, GLSL, HLSL, ISPC, Slang and Objective-C -- the list resolve_func_name_c_family uses: <leading cv-qualifiers, source order> <base type, verbatim> <one space><markers, outermost first> so `const char *`, `char **`, `Text &`, `Text &&`, `Text *&`, `volatile unsigned long *`; a pointer-level qualifier follows its star (`char *const *`); east-const `char const *` normalises to `const char *`. With no qualifier and no marker the old text is returned unchanged. It measures, then writes into one exact-size arena allocation; the declarator walk is a plain child chain, no recursion. def.return_types is untouched. Two deliberate differences from the upstream hunk. Upstream copied every type_qualifier node, which in these grammars includes constexpr, _Noreturn, mutable and __extension__ -- `constexpr int f()` would have become `constexpr int`. Only const, volatile, restrict, __restrict, __restrict__ and _Atomic are kept, and both cases are asserted unchanged. And the walk stops at the first declarator that is neither pointer nor reference, so `int (*f(void))(int)` stays `int` rather than becoming a wrong `int *`. RED, production reverted and the final tests kept: c_function_return_type_preserves_pointer_and_qualifier FAIL tests/test_extraction.c:1005: "char" != "const char *" cpp_method_return_type_preserves_pointer_and_qualifier FAIL tests/test_extraction.c:1072: "char" != "const char *" 351 passed, 2 failed c_function_return_type_plain_unchanged is the control: it passes before and after. GREEN: extraction 353, c_lsp 762, lang_contract 41, pipeline 281. Effect on real code -- Redis, production binaries, old against new. 39,132 nodes in both. 1,442 Function nodes change, in return_type only: void -> void * (270), char -> const char * (101), unsigned char -> unsigned char * (91), char -> char * (81), robj -> robj * (57), ... Functions with a `*` in return_type: 0 -> 1,440. With one worker the complete edge set apart from SEMANTICALLY_RELATED is byte-identical old against new across six runs (three each) -- CALLS, USAGE, WRITES, IMPORTS and the rest do not move. SEMANTICALLY_RELATED goes 350 -> 325 because that pass tokenises the return-type string into its type vector. One thing measured rather than assumed. With several workers, which `dict.h` Redis' src/dict.c imports (src/ or deps/hiredis/) depends on worker merge order. That is main's defect, not this change's: unmodified main picked src/dict.h in 2 of 10 runs and the hiredis header in 8; with one worker both binaries always pick src/dict.h. This change picked src/dict.h in 6 of 10. At ten runs a side that gap is not separable from chance (Fisher exact, two-sided p = 0.17), and a shifted rate would be unsurprising for a timing-dependent pick; it is recorded here rather than explained. The include-target fix is a separate change (DeusData#2227). Known limits, all pre-existing and out of scope: struct and class FIELDS have the same defect (`const char *data;` publishes char); C prototypes and declaration-only C++ methods are not extracted as definitions, so they are not reached; where a macro is misparsed as the type (`LUA_API const char *f()`), the base was already wrong and now reads `const LUA_API *`; the resolver's parser does not yet understand a pointer-level qualifier or a leading _Atomic/restrict and falls back to a named-type lookup for those, as it effectively did for every pointer return before. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
timothybrush
pushed a commit
to timothybrush/codebase-memory-mcp
that referenced
this pull request
Sep 24, 2026
Distilled from DeusData#1245 by Andrew Hundt (d15071c, the match_qn half). match_qn was a raw strstr(qn, library_id). Short ids fired inside unrelated words, and a service kind REPLACES the plain CALLS edge, so a false match both invents a service edge and deletes a real call: "gin." in plugin. -> route registration; any "/"-prefixed first argument then minted a Route node "nconf" in encryptionconfig -> CONFIGURES for every call into the package "tonic" in Monotonic -> gRPC; unparseable, so NO edge at all "express" in expression -> route registration "get_env" in get_envelope, get_environ -> CONFIGURES "dio" "surf" "curl" "resty" "phin" "hyper" "treq" "rocket" in studio, surface, curly, restyle, dolphin, hyperbolic, streq, rocketmq The rule (qn_hit_on_boundary; every occurrence of an id is tried, not only the first). A separator is any character that is not an ASCII letter or digit. before: start of string | separator | the id itself starts with a separator ("@trpc/server") | the id starts with an uppercase letter -- a capital opens a new CamelCase word whatever precedes it (AsyncHttpClient, IHttpClientFactory, NSURLSession) after: end of string | separator | the id itself ends in a separator ("gin.") | uppercase letter (GuzzleHttp, FeignClient, KafkaProducer) | digit (urllib2, Mint.HTTP2, amqp091-go) Rejected: a lowercase- or digit-initial id glued to a preceding letter or digit, and any id continued by a lowercase letter. Two deliberate differences from the upstream hunk: upstream's before-rule needed a GuzzleHttp table entry and would have dropped IHttpClientFactory and NSURLSession, and without the digit after-boundary urllib2 and amqp091-go stop matching. All three match on main today. No recall is traded for this. No boundary rule can tell "grequests" from "myrequests", so a library whose own name glues a prefix or suffix onto another id gets an explicit entry with the kind (and broker) it had on main -- 47 of them, found mechanically by running main's matcher and the new one over 173 candidate names and listing every one that went from a kind to NONE: HTTP grequests txrequests redaxios gaxios libcurl curlpp curlcpp hyperlocal guzzlehttp ASYNC aiokafka pykafka rskafka librdkafka rdkafkacpp aioamqp amqpstorm pamqp pyamqp amqprs amqpcpp pynats jnats aiomqtt amqtt hbmqtt umqtt mqttools emqtt rumqtt gomqtt libmosquitto mosquittopp pubsublite gocelery CONFIG wgetenv getenvb qgetenv dotenvx phpdotenv ROUTE_REG apiflask honox fasthttprouter GRAPHQL gqlparser gqlgenc aiogqlc Left at NONE on purpose: names that are not clients of the id they contain and were misclassified on main -- hypercorn, hyperlink, hyperopt, openresty, gqlalchemy, celeryconfig, kafkacat, gnatsd, pypubsub. One visible reclassification: hyperium/tonic was HTTP through "hyper" inside "hyperium" and is now gRPC through "tonic". RED, production reverted and the final tests kept: infrascan_service_pattern_match_rejects_ids_inside_words FAIL tests/test_infrascan.c:105: svc_case_mismatches(cases) == 28, expected 0 == 0 4 passed, 1 failed All 28 negatives mismatch on main. The 47 glued-library positives are RED the other way round: against the boundary rule without the entries. Each positive uses a name containing no second id, so it binds to its own entry -- "aiokafka.AIOKafkaProducer" would have passed through "KafkaProducer" and proven nothing. GREEN: infrascan 5, pipeline 281, parallel 74, extraction 350, edge_types_probe 59, route_canon 11, lang_contract 41, cross_repo 8, registry 64, mcp 318 (4 Windows-only skips). 0 failed. Effect on real code -- production binaries, main 92abefa against this change. main against main differs by 0 edges and 0 nodes on all three corpora, so every delta below is the change: django CONFIGURES 76 -> 74; both calls return as CALLS (geom.get_envelope, WSGIRequestHandler.get_environ) typescript CONFIGURES 12,026 -> 12,024; one bogus Route "/*type*/" gone with its CALLS and HANDLES; 3 calls return as CALLS kubernetes CONFIGURES 7,962 -> 7,609: all 353 are "nconf" inside encryptionconfig / authorizationconfig / authenticationconfig, and all 353 return as CALLS. 8 bogus Routes gone -- os.ReadFile of the serviceaccount namespace file, t.Run("/TwoWay"), "/tmp/test" -- with 26 CALLS, 2 HANDLES and 21 TESTS edges that pointed at them. 374 plain CALLS restored; 8 of them (validate.Monotonic and friends) had NO edge at all on main. GRPC_CALLS 60 -> 60. HTTP_CALLS 398 -> 407: nine calls main swallowed as route registrations now reach the existing arg_url heuristic like every other package. 88 of the 408 removed service-derived edges were read at the call site -- all of django and typescript, 82 of 402 in kubernetes: 88 false positives, 0 true positives lost. None of the three corpora uses a glued-name library, so the 47 entries are covered by the unit test only. Known limits: "_" is a separator, so optical_fiber.len still matches "fiber." exactly as requests_get matches "requests" by design; a lowercase id followed by a capital matches (kafkaProducer), which also admits a dioXide-style name; a user wrapper glued without a separator (mygetenv) is no longer classified; a glued-name library missing from the list is NONE until someone adds it. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
daidaiJ
pushed a commit
to daidaiJ/codebase-memory-mcp
that referenced
this pull request
Sep 29, 2026
… qualifiers Distilled from DeusData#1245 by Andrew Hundt (c51e5cb, the extract_defs.c half; the mcp.c and token-reduction parts are not taken). For C-family functions return_type was the text of the declaration's `type` node alone. In these grammars the `*`, `&` and `&&` live on the declarator and cv-qualifiers are sibling nodes, so const char *get_name(void) published char robj **lookupKeys(...) published robj Text &operator=(const Text &) published Text and no C or C++ function in any index had a pointer return type at all. The C/C++ resolver's return-type parser already strips a leading const/volatile and handles trailing `*`, `&`, `&&` -- branches that were unreachable for functions, so every `Foo *f()` was registered as returning Foo by value. c_declared_return_type() renders the declared type at both sites (functions and methods) for C, C++, CUDA, GLSL, HLSL, ISPC, Slang and Objective-C -- the list resolve_func_name_c_family uses: <leading cv-qualifiers, source order> <base type, verbatim> <one space><markers, outermost first> so `const char *`, `char **`, `Text &`, `Text &&`, `Text *&`, `volatile unsigned long *`; a pointer-level qualifier follows its star (`char *const *`); east-const `char const *` normalises to `const char *`. With no qualifier and no marker the old text is returned unchanged. It measures, then writes into one exact-size arena allocation; the declarator walk is a plain child chain, no recursion. def.return_types is untouched. Two deliberate differences from the upstream hunk. Upstream copied every type_qualifier node, which in these grammars includes constexpr, _Noreturn, mutable and __extension__ -- `constexpr int f()` would have become `constexpr int`. Only const, volatile, restrict, __restrict, __restrict__ and _Atomic are kept, and both cases are asserted unchanged. And the walk stops at the first declarator that is neither pointer nor reference, so `int (*f(void))(int)` stays `int` rather than becoming a wrong `int *`. RED, production reverted and the final tests kept: c_function_return_type_preserves_pointer_and_qualifier FAIL tests/test_extraction.c:1005: "char" != "const char *" cpp_method_return_type_preserves_pointer_and_qualifier FAIL tests/test_extraction.c:1072: "char" != "const char *" 351 passed, 2 failed c_function_return_type_plain_unchanged is the control: it passes before and after. GREEN: extraction 353, c_lsp 762, lang_contract 41, pipeline 281. Effect on real code -- Redis, production binaries, old against new. 39,132 nodes in both. 1,442 Function nodes change, in return_type only: void -> void * (270), char -> const char * (101), unsigned char -> unsigned char * (91), char -> char * (81), robj -> robj * (57), ... Functions with a `*` in return_type: 0 -> 1,440. With one worker the complete edge set apart from SEMANTICALLY_RELATED is byte-identical old against new across six runs (three each) -- CALLS, USAGE, WRITES, IMPORTS and the rest do not move. SEMANTICALLY_RELATED goes 350 -> 325 because that pass tokenises the return-type string into its type vector. One thing measured rather than assumed. With several workers, which `dict.h` Redis' src/dict.c imports (src/ or deps/hiredis/) depends on worker merge order. That is main's defect, not this change's: unmodified main picked src/dict.h in 2 of 10 runs and the hiredis header in 8; with one worker both binaries always pick src/dict.h. This change picked src/dict.h in 6 of 10. At ten runs a side that gap is not separable from chance (Fisher exact, two-sided p = 0.17), and a shifted rate would be unsurprising for a timing-dependent pick; it is recorded here rather than explained. The include-target fix is a separate change (DeusData#2227). Known limits, all pre-existing and out of scope: struct and class FIELDS have the same defect (`const char *data;` publishes char); C prototypes and declaration-only C++ methods are not extracted as definitions, so they are not reached; where a macro is misparsed as the type (`LUA_API const char *f()`), the base was already wrong and now reads `const LUA_API *`; the resolver's parser does not yet understand a pointer-level qualifier or a leading _Atomic/restrict and falls back to a named-type lookup for those, as it effectively did for every pointer return before. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
daidaiJ
pushed a commit
to daidaiJ/codebase-memory-mcp
that referenced
this pull request
Sep 29, 2026
Distilled from DeusData#1245 by Andrew Hundt (d15071c, the match_qn half). match_qn was a raw strstr(qn, library_id). Short ids fired inside unrelated words, and a service kind REPLACES the plain CALLS edge, so a false match both invents a service edge and deletes a real call: "gin." in plugin. -> route registration; any "/"-prefixed first argument then minted a Route node "nconf" in encryptionconfig -> CONFIGURES for every call into the package "tonic" in Monotonic -> gRPC; unparseable, so NO edge at all "express" in expression -> route registration "get_env" in get_envelope, get_environ -> CONFIGURES "dio" "surf" "curl" "resty" "phin" "hyper" "treq" "rocket" in studio, surface, curly, restyle, dolphin, hyperbolic, streq, rocketmq The rule (qn_hit_on_boundary; every occurrence of an id is tried, not only the first). A separator is any character that is not an ASCII letter or digit. before: start of string | separator | the id itself starts with a separator ("@trpc/server") | the id starts with an uppercase letter -- a capital opens a new CamelCase word whatever precedes it (AsyncHttpClient, IHttpClientFactory, NSURLSession) after: end of string | separator | the id itself ends in a separator ("gin.") | uppercase letter (GuzzleHttp, FeignClient, KafkaProducer) | digit (urllib2, Mint.HTTP2, amqp091-go) Rejected: a lowercase- or digit-initial id glued to a preceding letter or digit, and any id continued by a lowercase letter. Two deliberate differences from the upstream hunk: upstream's before-rule needed a GuzzleHttp table entry and would have dropped IHttpClientFactory and NSURLSession, and without the digit after-boundary urllib2 and amqp091-go stop matching. All three match on main today. No recall is traded for this. No boundary rule can tell "grequests" from "myrequests", so a library whose own name glues a prefix or suffix onto another id gets an explicit entry with the kind (and broker) it had on main -- 47 of them, found mechanically by running main's matcher and the new one over 173 candidate names and listing every one that went from a kind to NONE: HTTP grequests txrequests redaxios gaxios libcurl curlpp curlcpp hyperlocal guzzlehttp ASYNC aiokafka pykafka rskafka librdkafka rdkafkacpp aioamqp amqpstorm pamqp pyamqp amqprs amqpcpp pynats jnats aiomqtt amqtt hbmqtt umqtt mqttools emqtt rumqtt gomqtt libmosquitto mosquittopp pubsublite gocelery CONFIG wgetenv getenvb qgetenv dotenvx phpdotenv ROUTE_REG apiflask honox fasthttprouter GRAPHQL gqlparser gqlgenc aiogqlc Left at NONE on purpose: names that are not clients of the id they contain and were misclassified on main -- hypercorn, hyperlink, hyperopt, openresty, gqlalchemy, celeryconfig, kafkacat, gnatsd, pypubsub. One visible reclassification: hyperium/tonic was HTTP through "hyper" inside "hyperium" and is now gRPC through "tonic". RED, production reverted and the final tests kept: infrascan_service_pattern_match_rejects_ids_inside_words FAIL tests/test_infrascan.c:105: svc_case_mismatches(cases) == 28, expected 0 == 0 4 passed, 1 failed All 28 negatives mismatch on main. The 47 glued-library positives are RED the other way round: against the boundary rule without the entries. Each positive uses a name containing no second id, so it binds to its own entry -- "aiokafka.AIOKafkaProducer" would have passed through "KafkaProducer" and proven nothing. GREEN: infrascan 5, pipeline 281, parallel 74, extraction 350, edge_types_probe 59, route_canon 11, lang_contract 41, cross_repo 8, registry 64, mcp 318 (4 Windows-only skips). 0 failed. Effect on real code -- production binaries, main 92abefa against this change. main against main differs by 0 edges and 0 nodes on all three corpora, so every delta below is the change: django CONFIGURES 76 -> 74; both calls return as CALLS (geom.get_envelope, WSGIRequestHandler.get_environ) typescript CONFIGURES 12,026 -> 12,024; one bogus Route "/*type*/" gone with its CALLS and HANDLES; 3 calls return as CALLS kubernetes CONFIGURES 7,962 -> 7,609: all 353 are "nconf" inside encryptionconfig / authorizationconfig / authenticationconfig, and all 353 return as CALLS. 8 bogus Routes gone -- os.ReadFile of the serviceaccount namespace file, t.Run("/TwoWay"), "/tmp/test" -- with 26 CALLS, 2 HANDLES and 21 TESTS edges that pointed at them. 374 plain CALLS restored; 8 of them (validate.Monotonic and friends) had NO edge at all on main. GRPC_CALLS 60 -> 60. HTTP_CALLS 398 -> 407: nine calls main swallowed as route registrations now reach the existing arg_url heuristic like every other package. 88 of the 408 removed service-derived edges were read at the call site -- all of django and typescript, 82 of 402 in kubernetes: 88 false positives, 0 true positives lost. None of the three corpora uses a glued-name library, so the 47 entries are covered by the unit test only. Known limits: "_" is a separator, so optical_fiber.len still matches "fiber." exactly as requests_get matches "requests" by design; a lowercase id followed by a capital matches (kafkaProducer), which also admits a dioXide-style name; a user wrapper glued without a separator (mygetenv) is no longer classified; a glued-name library missing from the list is NONE until someone adds it. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
daidaiJ
pushed a commit
to daidaiJ/codebase-memory-mcp
that referenced
this pull request
Sep 29, 2026
… qualifiers Distilled from DeusData#1245 by Andrew Hundt (c51e5cb, the extract_defs.c half; the mcp.c and token-reduction parts are not taken). For C-family functions return_type was the text of the declaration's `type` node alone. In these grammars the `*`, `&` and `&&` live on the declarator and cv-qualifiers are sibling nodes, so const char *get_name(void) published char robj **lookupKeys(...) published robj Text &operator=(const Text &) published Text and no C or C++ function in any index had a pointer return type at all. The C/C++ resolver's return-type parser already strips a leading const/volatile and handles trailing `*`, `&`, `&&` -- branches that were unreachable for functions, so every `Foo *f()` was registered as returning Foo by value. c_declared_return_type() renders the declared type at both sites (functions and methods) for C, C++, CUDA, GLSL, HLSL, ISPC, Slang and Objective-C -- the list resolve_func_name_c_family uses: <leading cv-qualifiers, source order> <base type, verbatim> <one space><markers, outermost first> so `const char *`, `char **`, `Text &`, `Text &&`, `Text *&`, `volatile unsigned long *`; a pointer-level qualifier follows its star (`char *const *`); east-const `char const *` normalises to `const char *`. With no qualifier and no marker the old text is returned unchanged. It measures, then writes into one exact-size arena allocation; the declarator walk is a plain child chain, no recursion. def.return_types is untouched. Two deliberate differences from the upstream hunk. Upstream copied every type_qualifier node, which in these grammars includes constexpr, _Noreturn, mutable and __extension__ -- `constexpr int f()` would have become `constexpr int`. Only const, volatile, restrict, __restrict, __restrict__ and _Atomic are kept, and both cases are asserted unchanged. And the walk stops at the first declarator that is neither pointer nor reference, so `int (*f(void))(int)` stays `int` rather than becoming a wrong `int *`. RED, production reverted and the final tests kept: c_function_return_type_preserves_pointer_and_qualifier FAIL tests/test_extraction.c:1005: "char" != "const char *" cpp_method_return_type_preserves_pointer_and_qualifier FAIL tests/test_extraction.c:1072: "char" != "const char *" 351 passed, 2 failed c_function_return_type_plain_unchanged is the control: it passes before and after. GREEN: extraction 353, c_lsp 762, lang_contract 41, pipeline 281. Effect on real code -- Redis, production binaries, old against new. 39,132 nodes in both. 1,442 Function nodes change, in return_type only: void -> void * (270), char -> const char * (101), unsigned char -> unsigned char * (91), char -> char * (81), robj -> robj * (57), ... Functions with a `*` in return_type: 0 -> 1,440. With one worker the complete edge set apart from SEMANTICALLY_RELATED is byte-identical old against new across six runs (three each) -- CALLS, USAGE, WRITES, IMPORTS and the rest do not move. SEMANTICALLY_RELATED goes 350 -> 325 because that pass tokenises the return-type string into its type vector. One thing measured rather than assumed. With several workers, which `dict.h` Redis' src/dict.c imports (src/ or deps/hiredis/) depends on worker merge order. That is main's defect, not this change's: unmodified main picked src/dict.h in 2 of 10 runs and the hiredis header in 8; with one worker both binaries always pick src/dict.h. This change picked src/dict.h in 6 of 10. At ten runs a side that gap is not separable from chance (Fisher exact, two-sided p = 0.17), and a shifted rate would be unsurprising for a timing-dependent pick; it is recorded here rather than explained. The include-target fix is a separate change (DeusData#2227). Known limits, all pre-existing and out of scope: struct and class FIELDS have the same defect (`const char *data;` publishes char); C prototypes and declaration-only C++ methods are not extracted as definitions, so they are not reached; where a macro is misparsed as the type (`LUA_API const char *f()`), the base was already wrong and now reads `const LUA_API *`; the resolver's parser does not yet understand a pointer-level qualifier or a leading _Atomic/restrict and falls back to a named-type lookup for those, as it effectively did for every pointer return before. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
daidaiJ
pushed a commit
to daidaiJ/codebase-memory-mcp
that referenced
this pull request
Sep 29, 2026
Distilled from DeusData#1245 by Andrew Hundt (d15071c, the match_qn half). match_qn was a raw strstr(qn, library_id). Short ids fired inside unrelated words, and a service kind REPLACES the plain CALLS edge, so a false match both invents a service edge and deletes a real call: "gin." in plugin. -> route registration; any "/"-prefixed first argument then minted a Route node "nconf" in encryptionconfig -> CONFIGURES for every call into the package "tonic" in Monotonic -> gRPC; unparseable, so NO edge at all "express" in expression -> route registration "get_env" in get_envelope, get_environ -> CONFIGURES "dio" "surf" "curl" "resty" "phin" "hyper" "treq" "rocket" in studio, surface, curly, restyle, dolphin, hyperbolic, streq, rocketmq The rule (qn_hit_on_boundary; every occurrence of an id is tried, not only the first). A separator is any character that is not an ASCII letter or digit. before: start of string | separator | the id itself starts with a separator ("@trpc/server") | the id starts with an uppercase letter -- a capital opens a new CamelCase word whatever precedes it (AsyncHttpClient, IHttpClientFactory, NSURLSession) after: end of string | separator | the id itself ends in a separator ("gin.") | uppercase letter (GuzzleHttp, FeignClient, KafkaProducer) | digit (urllib2, Mint.HTTP2, amqp091-go) Rejected: a lowercase- or digit-initial id glued to a preceding letter or digit, and any id continued by a lowercase letter. Two deliberate differences from the upstream hunk: upstream's before-rule needed a GuzzleHttp table entry and would have dropped IHttpClientFactory and NSURLSession, and without the digit after-boundary urllib2 and amqp091-go stop matching. All three match on main today. No recall is traded for this. No boundary rule can tell "grequests" from "myrequests", so a library whose own name glues a prefix or suffix onto another id gets an explicit entry with the kind (and broker) it had on main -- 47 of them, found mechanically by running main's matcher and the new one over 173 candidate names and listing every one that went from a kind to NONE: HTTP grequests txrequests redaxios gaxios libcurl curlpp curlcpp hyperlocal guzzlehttp ASYNC aiokafka pykafka rskafka librdkafka rdkafkacpp aioamqp amqpstorm pamqp pyamqp amqprs amqpcpp pynats jnats aiomqtt amqtt hbmqtt umqtt mqttools emqtt rumqtt gomqtt libmosquitto mosquittopp pubsublite gocelery CONFIG wgetenv getenvb qgetenv dotenvx phpdotenv ROUTE_REG apiflask honox fasthttprouter GRAPHQL gqlparser gqlgenc aiogqlc Left at NONE on purpose: names that are not clients of the id they contain and were misclassified on main -- hypercorn, hyperlink, hyperopt, openresty, gqlalchemy, celeryconfig, kafkacat, gnatsd, pypubsub. One visible reclassification: hyperium/tonic was HTTP through "hyper" inside "hyperium" and is now gRPC through "tonic". RED, production reverted and the final tests kept: infrascan_service_pattern_match_rejects_ids_inside_words FAIL tests/test_infrascan.c:105: svc_case_mismatches(cases) == 28, expected 0 == 0 4 passed, 1 failed All 28 negatives mismatch on main. The 47 glued-library positives are RED the other way round: against the boundary rule without the entries. Each positive uses a name containing no second id, so it binds to its own entry -- "aiokafka.AIOKafkaProducer" would have passed through "KafkaProducer" and proven nothing. GREEN: infrascan 5, pipeline 281, parallel 74, extraction 350, edge_types_probe 59, route_canon 11, lang_contract 41, cross_repo 8, registry 64, mcp 318 (4 Windows-only skips). 0 failed. Effect on real code -- production binaries, main 92abefa against this change. main against main differs by 0 edges and 0 nodes on all three corpora, so every delta below is the change: django CONFIGURES 76 -> 74; both calls return as CALLS (geom.get_envelope, WSGIRequestHandler.get_environ) typescript CONFIGURES 12,026 -> 12,024; one bogus Route "/*type*/" gone with its CALLS and HANDLES; 3 calls return as CALLS kubernetes CONFIGURES 7,962 -> 7,609: all 353 are "nconf" inside encryptionconfig / authorizationconfig / authenticationconfig, and all 353 return as CALLS. 8 bogus Routes gone -- os.ReadFile of the serviceaccount namespace file, t.Run("/TwoWay"), "/tmp/test" -- with 26 CALLS, 2 HANDLES and 21 TESTS edges that pointed at them. 374 plain CALLS restored; 8 of them (validate.Monotonic and friends) had NO edge at all on main. GRPC_CALLS 60 -> 60. HTTP_CALLS 398 -> 407: nine calls main swallowed as route registrations now reach the existing arg_url heuristic like every other package. 88 of the 408 removed service-derived edges were read at the call site -- all of django and typescript, 82 of 402 in kubernetes: 88 false positives, 0 true positives lost. None of the three corpora uses a glued-name library, so the 47 entries are covered by the unit test only. Known limits: "_" is a separator, so optical_fiber.len still matches "fiber." exactly as requests_get matches "requests" by design; a lowercase id followed by a capital matches (kafkaProducer), which also admits a dioXide-style name; a user wrapper glued without a separator (mygetenv) is no longer classified; a glued-name library missing from the list is NONE until someone adds it. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 1, 2026
Distilled from #1245 by Andrew Hundt (upstream b835b5e), re-derived on main's cypher.c rather than cherry-picked -- the file has moved ~9k lines since the branch point. The WHERE clause runs twice: once early, on seed rows where only the first alias is bound, to prune; and once late, when every alias is bound, to decide. The early pass used the same two-valued evaluator as the late one. That evaluator answers "true" for a comparison on an alias it cannot see -- the permissive default that makes the late pass work -- but NOT and XOR invert whatever they are given, so MATCH (a)-[:CALLS]->(b) WHERE NOT b.name CONTAINS "x" evaluated NOT(true) on every seed and pruned every row. Zero results, no error, for a query that is correct. The early pass now uses a three-valued evaluator. A condition on an alias that is not yet bound is UNKNOWN; NOT of UNKNOWN stays UNKNOWN; AND and OR short-circuit only on a definite operand; XOR with any UNKNOWN operand is UNKNOWN. Only a definite FALSE prunes. The late pass is unchanged and still gives the final two-valued answer, so nothing that was correct before changes. Three deliberate differences from the upstream diff: - the unbound-alias rule lives in one leaf, eval_condition_partial, so the expression-tree path and main's legacy flat AND/OR path cannot drift apart; - that leaf guards c->variable == NULL. On main a condition whose left side is a function of literals, coalesce(0, 1) >= 2, carries no variable at all; the upstream version would have passed NULL to strcmp; - the flat path keeps main's rule that an empty condition list is true. RED before the fix, all four with row_count == 0, expected 1: cypher_exec_where_not_on_relationship_target, cypher_exec_where_mixed_alias_and, ..._or, ..._xor. GREEN after: cypher 208 passed; cypher + mcp 526 passed, 4 skipped (the Windows-only skips). eval_expr_partial recurses and is added to CBM_RECURSION_WHITELIST. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 1, 2026
Distilled from #1245 by Andrew Hundt (a slice of d15071c), with a different rule than upstream chose. Python's unittest.mock is used as a decorator: @patch("subprocess.run") def test_runs(self, run): ... decorator_method_name maps a bare `patch` to the HTTP method PATCH, and try_route_from_decorator_call accepts a receiver-less decorator call. When extract_route_path_from_args then finds nothing path-shaped, the function falls back to "/". So every mock-decorated test function became a Route handler, PATCH "/", and every one of those was wrong. Upstream fixed it with a has_receiver heuristic. That still admits @mock.patch("os.getcwd") (it has a receiver) and would reject the bare framework decorators Litestar and BlackSheep use, @get("/x"). The rule here is about the arguments, not the callee: a decorator call that HAS arguments but none of them path-shaped is not a route. The "/" default survives only for a zero-argument call such as @app.route(), which is the one case where "/" is what the framework means. No test on main relied on the default for a call with non-path arguments; checked before changing it. RED before the fix: extract_python_mock_patch_is_not_route FAIL tests/test_extraction.c:4023: mocked->route_path is not NULL extract_python_bare_decorator_route_rules FAIL tests/test_extraction.c:4063: mocked->route_path is not NULL The second test also pins the two shapes that must keep working: bare @get("/health") still yields a GET route, and zero-argument @app.route() still yields "/" ANY -- those assertions passed before the fix and after. GREEN after: extraction 351 passed; extraction pipeline edge_types_probe route_canon cross_repo infrascan lang_contract 754 passed, with the CALLS-breadth contract at 54 languages / 0 failures. The single remaining red in both runs is extract_spill_round_trip_keeps_every_field refusing to spill onto a disk below its 10 GB floor -- this host, not this change. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 1, 2026
Distilled from #1245 by Andrew Hundt (d030c8f). cbm_mcp_get_int_arg read integer arguments with yyjson_is_int and yyjson_get_int, and the int cast inside get_int truncates. A client sending {"limit": 4294967297} -- 2^32 + 1 -- got back 1: the low word of the value, silently, as a bound. The same read serves limit and offset on the query tools, so an oversized page request became a page of one. The argument is now read as a 64-bit signed or unsigned integer and only accepted if it fits in int; anything outside the range takes the caller's default, exactly as a non-integer does. In-range values and non-integer JSON behave as before. One difference from the upstream hunk: the unsigned comparison casts INT_MAX explicitly rather than relying on the implicit conversion. RED before the fix: mcp_get_int_arg FAIL tests/test_mcp.c:1777: val == 1, expected 17 == 17 The test now covers 2^32 + 1, its negative, INT_MAX + 1 and INT64_MIN each returning a distinct default, and INT_MAX, INT_MIN and -7 passing through unchanged. GREEN after: mcp 318 passed, 4 skipped (the Windows-only skips); mcp cli daemon_application 692 passed, 0 failed. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 1, 2026
Distilled from #1245 by Andrew Hundt (a slice of 1ac3c13). The route-registration suffix table knew .add_api_websocket_route but not the decorator forms FastAPI and Starlette actually use: @router.websocket("/ws") @app.websocket_route("/events") so a WebSocket endpoint produced no Route node on either resolver path and was invisible to the graph. Two entries join the table, both recorded as ANY like the existing WebSocket entry -- a socket upgrade is not an HTTP verb. Both are needed: matching is a pure suffix compare and .route does not match ...websocket_route, whose tail is _route. Upstream also changed pass_calls.c, pass_parallel.c and pipeline_internal.h. None of that is taken: main already carries the #952 sequential unresolved-callee route fallback its July base lacked, so the table entries are the only missing piece. RED before the fix, each form recorded on its own (the harness stops at the first failing assert, so the two were swapped and re-run): pipeline_python_websocket_decorators_register_routes FAIL tests/test_pipeline.c:6163: count_nodes_named(s, project, "/ws") (0) not >= 1 (1) FAIL tests/test_pipeline.c:6163: count_nodes_named(s, project, "/events") (0) not >= 1 (1) The @app.get("/health") control above those lines passed before and after. GREEN after: pipeline 282 passed; pipeline parallel extraction edge_types_probe route_canon infrascan 779 passed, 0 failed. The test drives the sequential resolver (one file). The parallel resolver reads the same table through cbm_service_pattern_route_method and its suite is green; a CBM_WORKERS=4 twin of this test is a small follow-up. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 1, 2026
Distilled from #1245 by Andrew Hundt (a slice of 0370d52). detect_url_in_args in the parallel resolver runs for every resolved call with no callee gating. A first argument like "/new/file.txt" clears normalize_url_arg (it has a second slash and no whitespace) and then reaches cbm_service_pattern_is_http_route_literal, whose filesystem extension list knew about config, key and socket files but not about documents or logs. So open_file("/new/file.txt") read_file("/docs/guide.md") write_file("/data/app.log") each minted a Route node and an HTTP_CALLS edge -- the graph reported a web endpoint where the program opens a file. Five extensions join the list: .log .md .pdf .rst .txt, alphabetical order preserved. Two things the tests are careful about, because the obvious fixture would not have been RED. A bare Python builtin open(...) never resolves to an indexed target, so it never reaches the URL detector at all; the bug shows only for calls that resolve to a project symbol, which is what the fixture defines. And /var/... is already rejected by the filesystem ROOT list, so the fixture paths start with segments that list does not know, leaving the extension guard as the only thing that can reject them. RED before the fix: infrascan_http_route_literal_guard_rejects_document_extensions FAIL tests/test_infrascan.c:44: ASSERT(!(cbm_service_pattern_is_http_route_literal("/new/file.txt", "open"))) pipeline_arg_url_rejects_document_file_paths FAIL tests/test_pipeline.c:6652: count_nodes_named(s, project, "/new/file.txt") == 1, expected 0 == 0 The pipeline test keeps a positive control above that line -- a genuine requests.get("/api/items") still yields its Route node and HTTP_CALLS edge -- so the guard cannot pass by rejecting everything. GREEN after: infrascan 4 passed; pipeline 282 passed; extraction edge_types_probe route_canon 420 passed. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 1, 2026
…them Distilled from #1245 by Andrew Hundt: the two exclusive-reservation helpers and their two call sites, hand-ported onto main's install.ps1. Nothing else from that file is taken -- the PR's copy also carries an older TLS block that would re-break Windows 10 (#1856), so a wholesale take was not an option. Two paths the installer used were guessable, and one of them adopted whatever it found: $TmpDir = Join-Path $env:TEMP "cbm-install-$(Get-Random)" New-Item -ItemType Directory -Path $TmpDir -Force Get-Random is seeded per process and its space is small, and -Force on an existing directory is a no-op that returns it. A directory planted at a guessed name became the staging area -- and the owner-only DACL the installer applies is set only afterwards, on a directory someone else created. The new helper names the directory with a GUID and creates it with -ErrorAction Stop, so a collision is a retry, never an adoption. $InstallerTmp = "$InstallerDest.new" was a fixed path in the install directory, reservable by anyone who could write there first. The sibling is now reserved with FileMode.CreateNew and FileShare.None on an unguessable name, which is atomic, before anything is copied into it. One deliberate difference from the PR: the sibling reservation runs INSIDE the existing try, and the cleanup checks that a path was assigned. That block is documented as best-effort ("a failure here still leaves a working install"), and a reservation that exhausts its 32 attempts should be one more best-effort miss, not a script abort under ErrorActionPreference=Stop. Verified: pure ASCII (PS 5.1 reads a BOM-less .ps1 as ANSI), LF endings matching main, both old call sites gone from live code. NOT verified: this file was not parsed by PowerShell. There is no pwsh on the macOS host, the Windows VM is down, and CI's Windows guards only check that `update` prints the install.ps1 command -- they never execute it. Opened as a draft for that reason; it needs one PowerShell parse before it merges. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 4, 2026
Distilled from #1245 by Andrew Hundt (a slice of 1ac3c13). The route-registration suffix table knew .add_api_websocket_route but not the decorator forms FastAPI and Starlette actually use: @router.websocket("/ws") @app.websocket_route("/events") so a WebSocket endpoint produced no Route node on either resolver path and was invisible to the graph. Two entries join the table, both recorded as ANY like the existing WebSocket entry -- a socket upgrade is not an HTTP verb. Both are needed: matching is a pure suffix compare and .route does not match ...websocket_route, whose tail is _route. Upstream also changed pass_calls.c, pass_parallel.c and pipeline_internal.h. None of that is taken: main already carries the #952 sequential unresolved-callee route fallback its July base lacked, so the table entries are the only missing piece. RED before the fix, each form recorded on its own (the harness stops at the first failing assert, so the two were swapped and re-run): pipeline_python_websocket_decorators_register_routes FAIL tests/test_pipeline.c:6163: count_nodes_named(s, project, "/ws") (0) not >= 1 (1) FAIL tests/test_pipeline.c:6163: count_nodes_named(s, project, "/events") (0) not >= 1 (1) The @app.get("/health") control above those lines passed before and after. GREEN after: pipeline 282 passed; pipeline parallel extraction edge_types_probe route_canon infrascan 779 passed, 0 failed. The test drives the sequential resolver (one file). The parallel resolver reads the same table through cbm_service_pattern_route_method and its suite is green; a CBM_WORKERS=4 twin of this test is a small follow-up. Co-authored-by: Andrew Hundt <ATHundt@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Owner
|
@ahundt — a consolidated thank-you, now that the last two route distills from this branch have landed on main. Here is everything that has come out of your work so far:
In a couple of places we chose a different rule than your branch did. For example, #2254 checks the shape of the decorator's arguments instead of using One distill is still open: #2251 ( Thank you again for the depth of this work, for the honest numbers, and for your patience while we landed it piece by piece. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. What does this PR do?
Hey, thanks again for the cool project! This PR continues my earlier PR #151. This Aug 2 addresses your requests: the
api-consolidationbranch is now merged with upstream main. Two true merge commits bring in every upstream commit throughf0c65fb2a997(2026-07-29):d0f00de6bd6dcombines upstream560ad40d19b2with the previously verified headb42801f3f765and resolves all 14 textual conflicts, and41fd25cdd86bintegrates thef0c65fb2a997release-hardening wave. No history was rewritten; both parents are preserved in each merge. At that merge the branch was 0 commits behind upstream main, and every upstream commit throughf0c65fb2a997remains contained here, so CI can run. A third merge,a0f4380b6f18(2026-08-01), integrates every upstream commit throughd6be58ef9d43(2026-07-31). At the time of writing (2026-08-02 16:39 UTC,d6be58ef9d43) that is still the upstream main head, so the branch has caught main and is 0 commits behind. Every defect claim in section 4 was re-verified at that revision. Beyond the three merges, 37 follow-up commits fix defects found while validating the merged results (listed with their verification in each commit message). The merge also removed the 86 committed soak/memlab run-log files (1.3 MiB) reported separately as issue #1330; upstream has since removed them on main in66326c2c.The core claim is unchanged: upstream main's post-edit indexing is labeled incremental but returns a graph that differs from an independent clean rebuild in 3 of 3 repetitions on the measured build, that is, the graph it serves after an edit is incorrect, and upstream's own source comment documents that
FILE_CHANGES_WITHandDATA_FLOWSedges are not reproduced incrementally (pipeline_incremental.c:351-360atd6be58ef9d43). This PR makes incremental source-structure indexing exact, and in its default configuration indexes the same large-codebase edit 3.2× faster than the measured upstream build (569 vs 1,828 ms). After an edit the server names every deferred derived view (rank, routes, semantic edges, architecture), omits stale rank values where returning them would mislead, and withincremental_derived_results_refresh=at_publishproduces a graph identical to the clean rebuild in 3 of 3 repetitions while still beating its own full rebuild (8,568 vs 9,768 ms).The PR adds optional indexing of installed dependency sources with package identity, optional graph-derived search ranking, and a shorter default MCP tool list, and it extends the read-only Cypher subset, Git-snapshot change detection, and semantic analysis. One attribution correction to my previous description: the shared daemon is upstream's subsystem (
src/daemon/, merged here on 2026-07-23); this PR retains it and contributes fixes and hardening on it, listed in section 2. Since findings like these are easy to misread from descriptions alone, I would appreciate review against the code, tests, and published run reports rather than summaries.The full classic API, request schemas, and upstream 3D visualization are retained. Every major capability has a one-command switch and defaults can be trivially revised if preferred.
2. Capabilities and controls
Every setting is declared in one registry with its default, accepted values, and environment override:
CBM_CONFIG_REGISTRYinsrc/cli/cli.c:12997. Two commands control everything:The six preset bundles pair a tool surface (
streamlined-orclassic-) withautomatic-dependency-source-indexing-disabledor-enabled, plus therank-disabledandminimal-indexingbenchmark ablations (codebase-memory-mcp config preset listshows each name with its description; the apply transaction iscbm_config_apply_preset()incli.c:12925). When an environment override pins one of a preset's keys, apply returns status 1 rather than claiming an ineffective change.File names below are unique in the tree and live under
src/; each switch is the exact argument list forcodebase-memory-mcp, with the default in parentheses. Line numbers are at the branch head7433fee6bed3.cbm_pipeline_run_incremental()pipeline_incremental.c:2826config set incremental_reindex always(default
always;full_rebuildrebuilds atomically from scratch andfast_mode_indexes_onlyrestricts incremental reindexing to fast-mode indexes)at_publishrecomputes everything immediately and matches the clean rebuild exactlycbm_pipeline_run_incremental()pipeline_incremental.c:2826config set incremental_derived_results_refresh at_publishconfig set rank_refresh at_publish(default:
defer_all_incremental_reindexesfor both)auto_indexdefault at session start, suppress release notices for dev builds, initialize the active-image fingerprint before comparison, validate fingerprint-cache epochs and admit cross-volume caches, and initialize capped memory once in the daemon instead of in every CLI frontendmain_run_daemon_ctl()main.c:1728daemon start/daemon stop/daemon status(default: session-managed, started automatically)
MEMBER_OFreverse edge and a cross-language normalization pass that repairs extraction-miss cases, so classes participate in traversals and inherit rank from their members instead of staying disconnectedprocess_def()pass_definitions.c:279repair:
pass_normalize.czz_order_core, moved from ninth to firstcbm_pagerank_compute()pagerank.c:407config set rank_enabled false(default
true)index_dependenciesindexes named packages on demand, and when installed packages exceedauto_dep_limit(default 20, 0 = unlimited, max 10,000) the most-imported packages are selected, ranked by project import referencescbm_dep_auto_index_effective()depindex.c:984ranking:
depindex.c:583-616config set auto_index_deps true(default
false)_hidden_tools; calling_hidden_toolsreveals the full classic tool list live, no restart (upstream advertises all tools and curates only statically via startup--profileflags)cbm_mcp_tools_list_range()mcp.c:3429config set tool_mode classic(default
streamlined)formatoverride and an actionable invalid-format error; the merged head keeps upstream's UTF-8 output guard so quoting or truncation cannot split a multibyte charactercbm_mcp_response_format()mcp.c:2685config set default_response_format json(default
toon)auto_indexin
CBM_CONFIG_REGISTRYconfig set auto_index false(default
true; upstream defaults tofalseand skips with a log hint)WITH, and optional-match semantics; adds multi-key ordering, output caps, and schema-aware rejection of unsupported syntax; the tool description, parser, and executor share one definition, and the query schema is stable against incidental storage changescbm_cypher_execute_impl()cypher.c:9091cbm_pipeline_pass_semantic_edges()pass_semantic_edges.c:1783config set semantic_edges_enabled falseconfig set similarity_enabled false(default
truefor both)cbm_git_snapshot_read()git_snapshot.c:319config set githistory_enabled false(default
true)pass_route_nodes.c, run in both the full and incremental pipelines) and adds a dedicated pass that links each HTTP call site to its handler, fixing missing or misclassified linkscbm_pipeline_pass_httplinks()pass_httplinks.c:1375config set httplinks_enabled false(default
true)build_fingerprint_modein
CBM_CONFIG_REGISTRYconfig set build_fingerprint_mode always_rehash(default
cached_exact)3. Measured results
The headline numbers come from a 39-cell benchmark experiment: 13 configurations, each run 3 times, paired and interleaved across configurations to reduce timing drift, on production builds in isolated Linux Docker containers (clang-18,
-O2, 16 build jobs, 16 container CPUs, container--init, fresh case-owned repositories and caches, arm64). Every cell applies the same real C source edit to a large codebase, runs incremental indexing, and compares the resulting graph with a separate clean rebuild after the same edit; every configuration also passes 12 retrieval-evidence checks (4 post-edit retrieval probes across the 3 repetitions: the changed file, its source text, its graph node, and its architecture evidence). All 39 planned cells completed with zero missing, corrupt, duplicate, or unplanned cells. Docker Desktop exposed 7.75 GiB despite the requested 16 GiB limit; the largest observed candidate peak was about 1.6 GiB, so no run was memory-capacity limited.Two scope statements, up front. First, the measured upstream build is
aedb979f5570(upstream main as of 2026-06-28), pinned when the benchmark cohort was frozen; the upstream main now merged into this branch isd6be58ef9d43(2026-07-31). The graph-correctness defect reproduced on the measured comparator, and at current upstream maind6be58ef9d43upstream's source still documents non-reproduced incremental edges (pipeline_incremental.c:351-360, the file is unchanged sincef0c65fb2a997) with no parity verification mechanism. Second, this matrix ran at PR revision2c2de761a24e, so its timing ratios apply to the named revisions, edit, and machine; a second full 39-cell matrix has since re-measured the merged branch head7433fee6bed3against an upstream build atd6be58ef9d43(below). What the merged head does run is the full test suite, including the incremental-parity tests, at the final verified head listed in section 7.All timings are medians (p50) over the 3 repetitions. Repeated query p50 is the median latency of the repeated read-only graph queries the retrieval probes issue within each candidate's persistent MCP session, so it measures a warm service, not process startup. Response payload p50 is the exact default tool-response size (tokens are the recorded deterministic bytes/4 estimate). Peak RSS is the highest peak resident memory observed across the 3 repetitions. "Deferred views named" means the server explicitly names each optional derived view awaiting recomputation after the fast source-structure refresh, and the freshness-scoped comparison still checks every remaining canonical node, edge, property, and file hash against the independent rebuild. A clean rebuild of the same post-edit tree is the ground truth: a post-edit graph that differs from it is incorrect, and queries against it can return wrong answers.
aedb979f5570, defaultminimal-indexingpresetUpstream main's post-edit reindex is labeled incremental but returns an incorrect graph, one that differs from the ground-truth rebuild (speed bought with wrong answers), while this PR's default is both 3.2× faster on the edit (569 vs 1,828 ms) and exact on non-stale source structure. The minimal preset outputs a graph identical to the clean rebuild while indexing the edit 3.6× faster (508 vs 1,828 ms) and full-indexing 1.37× faster (5.136 vs 7.055 s) than the measured upstream build. Two metrics go the other way and I state them plainly: upstream's repeated read-only query latency is lower (6.2 vs 16.7 to 23.2 ms; this PR's replies carry first-response freshness and provenance context and are 29% smaller, 429 vs 604 bytes), and upstream's full index is faster than this PR's default configuration (7.055 vs 10.029 s) because the default enables optional capabilities upstream does not have: disabling semantic edges alone brings the full index to 6.659 s, and the rank, similarity, git-history, and route ablations measure 9.269 to 10.140 s. Upstream peak RSS was not recorded in this run, so no memory comparison against upstream is claimed.
Refresh-at-publish recomputes every global derived result (rank, semantic edges, architecture, routes) immediately after the edit; at 8,568 ms that is still faster than its own 9,768 ms full rebuild, which is the only other way to get them all fresh. It is opt-in; the default defers exactly that graph-wide work and names the deferred views until refreshed.
Automatic dependency indexing stays off by default so its cost stays out of the default first index: enabling it indexed this workload's six installed packages in about 27 s of dependency work at first index, which is work upstream cannot do, and after an edit, checking unchanged dependencies cost 13 ms.
Two earlier development checkpoints (
c9726c7fd9b7,0fcb7e109518) ran in the same matrix and quantify what the post-merge work fixed: their native configurations auto-indexed dependencies (about 20.5 to 20.8 s of the 43.8 s full index) and carried a since-removed startup regression that pushed repeated queries to 187 to 206 ms; the current default full-indexes in 10.0 s with 23.2 ms repeated queries, and both checkpoints also failed the canonical-graph gate that the current head passes.A second 39-cell matrix re-measured the merged result: same container design (13 configurations, 3 repetitions each, paired-interleaved, isolated Linux Docker containers, arm64, 16 container CPUs), run at branch head
7433fee6bed3(2026-08-02) with the upstream comparator freshly built atd6be58ef9d43, the upstream main head this branch merges. All 39 planned cells completed with zero missing, corrupt, duplicate, or unplanned cells. The mutation differs from the first matrix (a new isolated C source file is added instead of editing an existing one), so timings are not comparable between the two matrices. Upstream's default again served an incorrect post-mutation graph in 3 of 3 repetitions: it differs from the ground-truth rebuild of the same tree, so queries after the edit can return wrong answers, and upstream ships no mechanism that detects this. Every configuration of this PR passed its graph gate.d6be58ef9d43, defaultminimal-indexingpresetminimal-indexingis the measured Pareto frontier of this matrix: no passing, fully measured candidate in the same workload is at least as good on overall quality and every cost axis while strictly better on any. All speed comparisons below are how much faster this PR is than current main (d6be58ef9d43) in this matrix; a negative value means slower, and every negative is bought by a named capability gain. Post-mutation indexing: +578% (473 vs 3,208 ms) at upstream-equivalent capabilities and +513% (523 ms) at the default, roughly 6x either way, and correct where upstream's result is incorrect. Clean rebuild: -6% (8.025 vs 7.508 s) at upstream-equivalent capabilities (parity verification runs in the test and benchmark harness, so it adds nothing here; the difference is graph-building work this branch always does, including theMEMBER_OFclass-connectivity repair and exact statistics publication, with no single component isolated by an ablation, and at this price the post-mutation graph is verified correct where upstream's is incorrect); -13% (8.674 s) at the default, which adds ranking, similarity, git history, and route linking with the measured benefits in the table below; +81% (4.141 s) withminimal-indexing, fully correct. Repeated query: -2% (17.6 vs 17.3 ms) at upstream-equivalent capabilities, closing the first matrix's 3 to 4x gap, -14% (20.0 ms) at the default, +4% (16.6 ms) withminimal-indexing, at effectively equal peak memory (1,532 vs 1,536 MB). In this workload upstream's default reply is smaller (306 vs 407 bytes); the PR's replies carry the first-response freshness and provenance context described above. Refresh-at-publish again produced the fully identical graph while beating its own clean rebuild (7,805 ms vs 8,437 ms).The default's remaining costs are itemized capability prices, not overhead, and each is refundable with one switch. Overhead, cost that returns no value, is what this head (
7433fee6bed3) removed: the first matrix's repeated-query deficit does not reproduce (17.6 ms with upstream-equivalent capabilities vs upstream's 17.3 ms), and the like-for-like clean rebuild is -6% (0.52 s, 8.025 vs 7.508 s) at equal peak memory. Parity verification runs in the test and benchmark harness, not in production indexing, so that 0.52 s is not a verification tax: it is graph-building work this branch always does, including theMEMBER_OFclass-connectivity repair and exact statistics publication, with no single component isolated by an ablation. What the build delivers at that price is the guarantee upstream does not offer at any price: the post-mutation result is verified correct. Every cost above that baseline is priced below against the same build with exactly that capability toggled (all other defaults unchanged); the default configuration's own rebuild spread is [8.437, 9.599 s] over 3 repetitions, so prices under 0.6 s are smaller than that spread and should be read as approximate:SEMANTICALLY_RELATEDpair matches the clean rebuild, none missing, none spuriousconfig set semantic_edges_enabled falseconfig set rank_enabled falseSIMILAR_TOedges for near-duplicate discoveryconfig set similarity_enabled falseconfig set httplinks_enabled falseFILE_CHANGES_WITHcoupling edges and Git-scoped incremental change detectionconfig set githistory_enabled falseconfig set auto_index_deps trueSummed, the default's entire repeated-query premium over upstream at the merged head (
7433fee6bed3) is 2.7 ms (20.0 vs 17.3 ms). The one axis where upstream is faster like-for-like is the clean rebuild, where this PR is -6%; what stands on the other side of that price is the post-mutation result: +578% (473 vs 3,208 ms) with a verified-correct graph, where upstream's faster-looking incremental serves an incorrect graph, one that differs from its own ground-truth rebuild in 3 of 3 repetitions, speed bought with wrong answers and no switch that refunds it.Result-quality contribution of the two new optional capabilities, isolated on one PR build (
13a4679b27f6) and test project by repeating the same index-then-query measurement 5 times per state, alternating the capability between enabled and disabled. MRR (Mean Reciprocal Rank) rewards placing the known correct answer near the top, Hit@1 and Hit@5 report whether it appears first or within the first five, nDCG@5 (normalized Discounted Cumulative Gain) rewards correct ordering within the first five, and task success means the task's named expected answer was returned:zz_order_core) ahead of eight lexical-only decoyszz_order_corereturned 1st of 9; MRR, Hit@1, Hit@5, nDCG@5, task success all 1.000zz_order_core9th of 9; MRR 0.111, the other four metrics 0canonicalDependencyAPI) with its dependency, package, and read-only provenance on the same resultcanonicalDependencyAPIreturned 1st, with package identitycanonicalDependencyAPIabsentSemantic refresh quality was re-verified in the same matrix: with refresh at publish, every judged
SEMANTICALLY_RELATEDpair matches the clean rebuild, with none missing and none spurious.4. Upstream main defects this PR corrects
Every claim in this table was re-verified by direct code reading at upstream main
d6be58ef9d43(2026-07-31, the revision merged into this branch) unless a different revision is named; upstream file:line references are at that revision.d6be58ef9d43cbm_pipeline_run_incremental()pipeline_incremental.cFILE_CHANGES_WITHandDATA_FLOWSedges are not reproduced incrementally (pipeline_incremental.c:351-360); no parity guarantee or verification mechanism exists; measured divergence 3 of 3 onaedb979f5570resolve_store()mcp.cresolve_store_internal,mcp.c:2081-2083); upstream's daemon single-ownership makes cross-process replacement rarer but the check is still absentWITHstages, optional matches, and multi-keyORDER BYcompute correctlycbm_cypher_execute_impl()cypher.cparse_order_by_clauseparses exactly one ordering key, so secondaryORDER BYkeys are silently dropped (cypher.c:1699-1709); the aggregate, post-WITH, and optional-match failures were verified on the measured comparatoraedb979f5570; upstream has since fixed aggregate-column classification and two optional-match defects (PRs #1221, #1177, #1385, contained here through thed6be58ef9d43merge), while multi-keyORDER BYremains absent upstreamLIMITlowers but cannot bypass the configured row cap, and truncation is reported with the limiting condition instead of being silentscan_pattern_nodes()cypher.cmax_rowstimes a fixed growth factor before inline property filters andWHERErun (cypher.c:3027-3052, cap applied atcypher.c:3037), so matching rows past the prefix are discardedauto_indexin
CBM_CONFIG_REGISTRYfalseand an unset value skips indexing with only a log hint (mcp.c:11156-11169)MEMBER_OFreverse edge and a cross-language repair pass for extraction-miss casesprocess_def()pass_definitions.c:279repair:
pass_normalize.cDEFINES_METHODonly when extraction supplies the parent; there is no reverse edge and no repair pass (pass_normalize.cdoes not exist upstream), so extraction-miss classes stay disconnectedindex_statusreports whether the working tree is dirty and whether the index matches itadd_git_context_json()mcp.cadd_git_context_json(mcp.c:2242) emits branch and commit identity but no working-tree-state field; upstream's watcher tracks a dirty signature internally but does not report it as provenancecbm_pipeline_githistory_compute_with_limits()pass_githistory.c:646FILE_CHANGES_WITHcandidates at a hard-coded 8,192 (MAX_COUPLINGS,pass_githistory.c:346) collected in hash-table iteration order with no sort, no configuration, and no log (pass_githistory.c:215-217, 331), so on a history with more qualifying pairs the emitted coupling edges are an arbitrary subset rather than the strongest couplings; per-file temporal facts stop at the first 16,384 files encountered (MAX_FILE_TEMPORAL,pass_githistory.c:406)Three defect claims from my previous description are retired because re-auditing upstream showed them already fixed there: the half-opened SQLite handle on failed open (closed upstream in
e6e9c58e, before the measured comparator), the config-linking pass's large stack arrays (the pass was rewritten upstream and now holds only small static tables), and the dropped route-discovery pass (rebuilt upstream aspass_route_nodes.c; the merged branch keeps upstream's pass and adds its own, section 2). Shutdown lifecycle also converged: upstream's daemon now joins worker threads and plumbs cancellation, and the merged branch carries both sides' shutdown work; this PR's remaining distinct contribution there is the verification discipline (leak gate, cancellation and reaping tests) rather than a defect claim.One further upstream hard-limit defect is fixed here but deliberately kept out of the table, because neither tree currently wires the affected scanner into a production pass: upstream's env-URL walker holds a fixed 512-byte-per-frame path stack and silently skips directories nested deeper than 256 levels (
pass_envscan.c:514, 532atd6be58ef9d43). This branch's walker uses dynamically grown path storage (walk_path.h), logs unrepresentable bindings instead of dropping them, and pins both behaviors with the regression testsenvscan_walks_more_than_256_pending_directoriesandenvscan_accepts_root_path_longer_than_512_bytes.Every remaining row is regression-tested. Defects found while merging upstream and validating the result were fixed before this update; each is documented in its commit message. Two behaviors are boundaries rather than defects: the Cypher subset does not claim full openCypher conformance, and optional derived results refresh on the documented deferred schedule. Production
src/contains nosystem()calls, and the heap-leak gate reports 0 leaked bytes at the final verified head.5. Compatibility, breaking changes, and migration
config set tool_mode classicrestores the classic list on a running installation. Streamlined and classic surfaces answer from the same graph and Cypher contract.daemon stoprefusal while committed clients remain, and orderly SIGINT/SIGTERM shutdown (cancel and reap work, join background threads, close transports, release claims and leases, remove the endpoint). This PR's daemon changes are the fixes listed in section 2.--reset-indexesis passed; the system-wide upgrade verified at an earlier head preserved all 462 existing indexes across 17 detected clients. The installer recognizes configuration written by earlier development builds and replaces it canonically instead of rejecting it, keeps user-owned TOML child tables when replacing a managed parent table, and refuses to overwrite genuinely user-edited entries.d6be58ef9d43, the repository's CI has run its full matrix on this pull request at7433fee6bed3: the test shards (Linux gcc, macOS 14 and 15-intel, Windows CLANG64), ThreadSanitizer on three platforms, lint, smoke on Linux, macOS, and Windows, CodeQL, and the security gates all pass; the DCO sign-off check is the only failing check.6. How to run the tests
The full suite passed with zero failures at the final verified head (7,823 passed, 2 platform-specific skips), and the CI lint gates (
lint-ci) exit 0 there. The suite includes generalized extraction and incremental tests across C, C++, Go, Python, JavaScript, TypeScript, Java, Kotlin, and Rust, mixed-language call resolution, the daemon lifecycle, bootstrap, and IPC suites, and the tests that arrived with the merged upstream commits. The benchmark runner records exact candidate commits, binary SHA-256 values, compiler and optimization flags, CPU and memory allocation, environment overrides, case cleanup, and invalid-run status in an immutable audit manifest.7. Evidence and revisions
f0c65fb2a997d6be58ef9d43d0f00de6bd6d560ad40d19b2and the previous verified headb42801f3f765; all 14 textual conflicts resolved, no history rewrite41fd25cdd86bf0c65fb2a997a0f4380b6f18d6be58ef9d43; no history rewrite24aa8a415f5c7433fee6bed32c2de761a24eaedb979f5570c9726c7fd9b7,0fcb7e10951813a4679b27f6All experiments ran on production builds. The first 39-cell run set's immutable audit manifest has SHA-256
1becd57c00edbe965073f4d7c9a79cca8812947f207ed51851c856369aa2c002, and the second run set's (branch head7433fee6bed3, 2026-08-02) has SHA-2561e8f0d96128bd352107a0fb1633bcb0bcbd696a5ea8cb8226569d62e580a65e0; each experiment root retains the input specification, per-run manifests, container environment records, logs, build identities, and cleanup status, so the numbers are independently reproducible with the command above. The generated reports are published in a gist; its current tree contains the 2026-07-28 run set and the earlier 39-cell runbe2097bdbf14, and the focused ranking, dependency, and semantic reports remain in the gist revision history.Ranking follows PageRank (Brin and Page, 1998) and LinkRank (Kim, Son, and Jeong, 2010); nDCG (Järvelin and Kekäläinen, 2002) defines the ranking metric; RepoGraph (ICLR 2025) motivates repository graphs. The repository experiments, not the citations, are the evidence that this implementation works.
8. Checklist
make -f Makefile.cbm testat the final verified head: 7,823 passed, 2 platform-specific skips)make -f Makefile.cbm lint-ciexit 0 at the final verified head: cppcheck 2.20, LLVM 20 clang-format, suppression-policy and source-safety checks; the previous body's 49 open findings and 20 formatting violations are cleared)