Skip to content

fix(fts): preserve exact top-level multimatch ties - #8702

Open
BubbleCal wants to merge 1 commit into
mainfrom
yang/fts-top-level-cross-column-multimatch
Open

fix(fts): preserve exact top-level multimatch ties#8702
BubbleCal wants to merge 1 commit into
mainfrom
yang/fts-top-level-cross-column-multimatch

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What is the bug?

Top-level cross-column MultiMatch pushes the global k into each field's Match child. The leaf path truncates score ties before final row IDs are resolved, so the final MAX(_score) aggregate and (score DESC, row_id ASC) sort can only reorder the wrong survivor set.

On the 10M-row MMLB direct workload, current main mismatched the exhaustive oracle in 121 of 500 cases and produced unstable timed signatures for the same boundary-tie queries.

Linear: https://linear.app/lancedb/issue/OSS-1602

How does this PR fix the problem?

  • Plans one bounded, exact single-column CompoundQueryExec per field.
  • Keeps field scoring independent and retains the existing bounded outer DisMax / MAX(_score) aggregation.
  • Plans each field independently: complete, unchanged indexed fields retain an exact bounded compound scorer.
  • Replans only partial-index, overlay-backed, missing-index, or otherwise ineligible fields through the unbounded exact leaf fallback before the final aggregate.
  • Preserves exhaustive execution for unbounded queries.
  • Supports both explicit MultiMatch and fieldless Match expansion.
  • Canonicalizes signed zero boosts to +0.0 before field expansion so MAX aggregation and total-order sorting agree.

The generic cross-column compound scorer was also prototyped for this root shape, but it paid row-address merge costs without a selective probe leaf. The field-local exact design was faster and keeps the final intermediate set bounded by fields * k.

Correctness validation

Environment: 10M-row MMLB dataset, two independently indexed text columns, c4-highmem-16, 8 workers, 64 GiB index cache.

Oracle case Current main This PR
k=10, 250 queries 120 mismatches 0 mismatches
k=100, 250 queries 1 mismatch 0 mismatches
Total 121 / 500 mismatches 500 / 500 exact

The target also had zero row/result signature instability across four timed processes.

This is primarily a correctness fix. Current main's higher throughput is not a valid performance baseline because its result set is incorrect on the affected queries.

Tests and checks

  • Added explicit and fieldless top-level cross-column MultiMatch oracle coverage.
  • Covers equal-score kth ties, reversed segment order, signed zero boosts, unbounded execution, and mixed bounded/exact planning with one partially covered field.
  • cargo check -p lance --tests
  • cargo fmt --all -- --check
  • git diff --check
  • Pre-commit fmt and typos checks

Per the requested workflow, I did not run local cargo test or cargo clippy; CI will run them.

@github-actions github-actions Bot added the bug Something isn't working label Aug 21, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 21, 2026
@BubbleCal
BubbleCal force-pushed the yang/fts-top-level-cross-column-multimatch branch from c581b76 to 10f53f1 Compare August 21, 2026 17:02
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 21, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Aug 21, 2026
@BubbleCal
BubbleCal force-pushed the yang/fts-top-level-cross-column-multimatch branch from 10f53f1 to 3d27cf9 Compare August 21, 2026 17:21
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Aug 21, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

Both prior findings are fixed. The revised planner preserves each eligible field’s exact bounded top-k scorer, confines exhaustive fallback to ineligible fields, and the mixed partial-coverage result matches the exhaustive oracle.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant