refactor(fts): add exact posting load policies - #8667
Merged
Conversation
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The policy cleanly separates grouped read-ahead from cold exact demand: existing callers keep their current behavior, while an exact lookup reuses a resident group or singleflights a singleton load. The implementation preserves query semantics and cache/metrics boundaries, with focused coverage for concurrency, coalescing, and warm-cache precedence. Production performance remains appropriately deferred to the first consumer.
lance-gatekeeper
Bot
added
the
K-approved
Xuanwo
approved these changes
Aug 20, 2026
BubbleCal
added a commit
that referenced
this pull request
Aug 21, 2026
## Feature
Each FTS index source scores in its own partition-local DocId domain.
Cross-column compound queries need to compose those scorers in one
globally ordered row-address domain without assuming independently built
indices share local document layouts.
This PR adds the cross-column scorer core:
- validates and caches strictly ordered row-address projections
- maps local scorer iteration, shallow bounds, and advances into
row-address order
- materializes an exact fallback for reordered projections
- rejects duplicate row-address mappings that could combine content from
different local documents
- lazily merges multiple physical sources for one semantic leaf
- preserves conservative bounds, competitive floors, and equal-score
row-address ties
- analyzes compound plans for positive generator coverage, cost,
feasibility, and conservative score bounds
- supports staged positive generation followed by candidate-scoped
required, optional, and prohibited probing
- performs one exact global top-k collection after composing Match,
Phrase, Boolean, MultiMatch, and Boost scorers
- uses candidate-scoped row-address and scoring-length reads when
staging is selective
The dataset Scanner does not call this core yet; that integration
remains in the final consumer PR.
## API and compatibility
There is no file-format change.
This PR exports the low-level async Rust entry point used by the later
`lance` planner integration:
```rust
pub async fn cross_column_compound_search(
columns: &[(String, Vec<Arc<InvertedIndex>>)],
query: &FtsQuery,
params: &FtsSearchParams,
prefilter: Arc<dyn PreFilter>,
metrics: Arc<dyn MetricsCollector>,
) -> Result<(Vec<u64>, Vec<f32>)>
```
The API requires a bounded limit, preserves exact `(score DESC,
row_address ASC)` ordering, and returns an error for unsupported or
internally inconsistent scorer state.
Existing `PartitionDocuments::resolve_addresses` and
`estimated_address_read_bytes` behavior remains unchanged from `main`,
so current production cache and I/O behavior is not altered before
Scanner integration.
## CI failure addressed
The previous head split scorer foundations from their production
consumer, leaving 43 groups of private items unused under `-D
dead-code`. This revision folds the cross-column core into the same PR
so those components have real production call paths.
Unused leaf-role metadata was deleted. No `allow(dead_code)`, test-only
gating, or visibility workaround was added.
## Scope boundary
This is PR 3 of the OSS-1603 stack and builds on #8666 and #8667, both
merged.
It does not include the previously deferred same-column delayed
`MUST_NOT` probing work from OSS-1705. Cross-column prohibited clauses
are supported as ordinary query semantics, but this PR does not change
the same-column `BooleanScorer` or its probing strategy.
## Validation
- `cargo fmt --all -- --check`
- `git diff --check origin/main...HEAD`
- deterministic projection tests for ordered, deleted, remapped,
duplicate, and out-of-order layouts
- materialized fallback and collision regressions
- merge-scorer ordering, lazy initialization, advance, shallow-bound,
floor, tie, and duplicate tests
- seeded randomized exhaustive oracle coverage for SUM, MAX, MUST,
required-optional, signed Boost, and nested prohibited shapes
- staged generator, candidate resolution, phrase, visibility, and
quantized-length tests
- static reachability audit covering every item reported by the failed
CI jobs
Per project workflow, GitHub CI is the authoritative Cargo test and
Clippy run for this revised head.
## Performance validation
No end-to-end performance benefit is claimed yet because the dataset
Scanner does not call this entry point in this PR. A main-vs-this-PR
dataset benchmark would execute the existing fallback path and measure
noise.
The final consumer PR will enable this core and report independent
warm/cold ABBA results with latency, throughput, CPU, bytes, requests,
cache metrics, and exact result digests.
## Stack
1. #8666 — WAND scoring and bound exactness (merged)
2. #8667 — posting loading and cache policy (merged)
3. **This PR:** row-address and cross-column compound scorer core
4. Dataset planner / execution integration and end-to-end benchmark
Part of
[OSS-1603](https://linear.app/lancedb/issue/OSS-1603/add-candidate-driven-execution-for-cross-column-boolean-fts-queries).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Other Changes
Read-ahead posting groups reduce storage requests and improve cache reuse, but a future candidate-driven consumer also needs a way to avoid reading neighboring token rows on a cold miss.
This PR adds the internal loading infrastructure for that choice:
PostingReadPolicy::{ReadAhead, CacheAwareExact}[token_id, token_id + 1)cache entry on a cold exact missBehavior and compatibility
There is no public API or file-format change.
The existing production entry point still selects
ReadAhead. TheCacheAwareExactconstructor is test-only in this layer and will be enabled by the later cross-column consumer PR, so merging this PR alone does not change production query behavior.A cache-only probe miss is intentionally omitted from the query-level
MetricsCollector; only the path that serves the posting records a query cache hit or miss. The lower-levelLanceCache::stats()still observes the probe.Scope boundary
This is PR 2 of the OSS-1603 stack and depends only on the exactness foundation merged in #8666.
It does not include the previously deferred same-column delayed
MUST_NOTprobing work from OSS-1705, and it does not add or change Boolean execution.Validation
cargo fmt --all -- --checkgit diff --check origin/main...HEADPer project workflow, GitHub CI is the authoritative Cargo test and Clippy run for this rebased head.
Performance validation
No end-to-end performance benefit is claimed for this layer because there is no production
CacheAwareExactcaller yet. A main-vs-PR2 query benchmark would execute the sameReadAheadpath on both builds and measure noise rather than this code.The first consumer PR will benchmark the policy independently with cold singleton-demand, cold same-group multi-demand, and warm/prewarmed ABBA cases, including bytes, requests, posting loads, cache metrics, and exact result digests.
Stack
Part of OSS-1603.