feat: indexed nearest-by join over Lance — Spark 4.2 SQL Catalyst

[sezruby]

Adds a single module, lance-spark-knn-4.2_2.13, that transparently accelerates Spark 4.2's native APPROX NEAREST k BY DISTANCE ... join (the NearestByJoin operator from SPARK-56395) over a Lance scan.

What it does

Spark's default rewrite lowers NearestByJoin to an O(|L|×|R|) cross-product + MaxMinByK. This module rewrites it onto a no-shuffle, per-partition native probe against a Lance vector index instead.

• IndexedNearestByJoinRule — a post-hoc resolution rule (the only injection point that sees the unrewritten NearestByJoin; Spark's own RewriteNearestByJoin runs first, in the FinishAnalysis batch) rewrites the operator into a single LanceKnnJoinLogicalPlan.
• LanceKnnJoinStrategy lowers that to LanceKnnJoinExec, which drives one mapPartitions → per-row LanceProbe (native ANN/exact search) → bounded top-K → late materialize by row address. No shuffle, no broadcast.
• Opt-in behind spark.lance.knn.indexedNearestByJoin.enabled (off by default). When disabled, the query falls through to Spark's built-in rewrite.
• Right-side WHERE is translated to a Lance prefilter, or the rewrite is refused when the predicate can't be pushed in full — never a silently dropped residual.
• Metrics: L2, cosine, dot. nprobes / refineFactor are tunable via spark.lance.knn.* configs.

The module carries its own probe core (LanceKnnJoinStage / LanceProbe / TopKHeap / Metric); it depends on lance-spark-base for LanceRuntime and on the Spark 4.2 connector at test scope.

Tests

• SQL end-to-end: parser → rule → strategy → exec against a real Lance dataset, oracle-checked — including WHERE prefilter pushdown and the opt-in gating.
• IVF-PQ approximate-recall suite driven through the same SQL path (builds a real index; checks recall floor and that refineFactor helps).
• Rule-unit suite (metric/direction matching, alias/filter unwrapping, prefilter translator) plus probe/heap unit tests.

Notes

• Not compiled in this environment — review / a reactor build is the gate.
• The fork's default CI matrix (spark.yml runs mvn test -pl lance-spark-<spark>_<scala> -am) builds the connector modules and their upstream deps only; this module is downstream, so it is not built by that matrix. Verifying it needs a reactor build (e.g. ./mvnw test -pl lance-spark-knn-4.2_2.13 -am).
• Scope is intentionally SQL-only. The manual df.kNearestJoin DataFrame API is deferred — it can be added later as a Spark-version-agnostic base module when there's demand.

🤖 Generated with Claude Code