Skip to content

test(merge_insert): cover the reordered full-schema duplicate-key path - #8637

Open
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/8282-rowaddr-order-fail-closed
Open

test(merge_insert): cover the reordered full-schema duplicate-key path#8637
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/8282-rowaddr-order-fail-closed

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Summary

Adds one regression test for the shape reported in #8282: an indexed merge_insert whose source carries every target field in a different order and repeats a join key.

That shape has no coverage today. #8509 fixed the misaligned target-key lookup behind #8280, and measurement shows the same change also fixes #8282, but the two regressions #8509 added both use a partial-schema source with a single row, so nothing guards the full-schema InsertAll duplicate-key path where the misread reached the row-address reconciler.

What it pins, and why the assertions look the way they do

Verified in both directions: on 526988981 (the #8509 merge) the test passes; reverting the offset lookup makes it fail with Merge insert update row address (0, 0) is missing from the target fragment; next target row address is (0, 1), which is exactly what #8282 reported.

Two preconditions are asserted rather than assumed, because the ambiguity rejection the test expects has several call sites and the v2 fast-path one emits a byte-identical message:

  • !can_use_create_plan(...) keeps the merge on the v1 slow path. Without it the duplicate is caught by the v2 writer and the reconciler is never involved.
  • matches!(check_compatible_schema(...), SchemaComparison::Subschema) pins the misclassification. A reordered full schema fails the order-sensitive full match and falls through to Subschema, which sets add_row_addr. That _rowaddr is what turns the misread into an internal error rather than silently dropped rows. Making the full match order-insensitive would route this source to RewriteRows, and a message-only assertion would still pass.

The negative check (!error.contains("is missing from the target fragment")) runs before the positive one so a reconciler regression is what gets reported.

The message is asserted instead of the Error variant deliberately. This error crosses the sort/repartition in update_fragments, so DataFusion returns it shared behind an Arc and the conversion in lance-core yields Execution or InvalidInput depending on how many partitions still hold a reference. The variant tracks the machine's CPU count; the text does not.

Validation

  • cargo test -p lance --lib test_indexed_full_schema_reordered_duplicate_keys_are_ambiguous
  • cargo test -p lance --lib dataset::write::merge_insert::tests (181 passed)
  • cargo clippy -p lance --tests -- -D warnings
  • cargo fmt --all -- --check

Scope

Test-only; no production code changes. If maintainers agree the coverage is right, #8282 can be closed as fixed by #8509.

One limit worth stating: the dataset_ops artifact from #8282 is not public, so this reproduces the reported mechanism rather than replaying that session. The evidence it is the same path is the reported op parameters (rotate: 215, dup_first: true, i.e. a rotated column order with a repeated first key), the identical failure point, and the identical error text before #8509. The report's storage version 2.0, stable row ids disabled, and longer preceding mutation history were not reproduced.

A full-schema source whose fields are merely reordered was classified as
a partial schema on the indexed slow path, and the target key columns
were then located by offsetting the source key positions. That offset
landed on a nullable payload column, so every matched row looked
unmatched and took the insert branch, which carries the real target
_rowaddr into the in-place column rewrite. A repeated key fed the same
address into the update stream twice and the reconciler reported a
row-address ordering violation (lance-format#8282) instead of the ambiguity the
caller has to act on.

PR lance-format#8509 fixes that lookup for lance-format#8280, and measurement shows it fixes
this shape too: before it the merge fails with "update row address
(0, 0) is missing from the target fragment", and after it the merge
fails with the ambiguous-key rejection naming id = 1. The two
regressions lance-format#8509 adds cover partial-schema sources with a single row,
so nothing guards the full-schema InsertAll duplicate-key path this test
exercises.

Both routing preconditions are asserted rather than assumed, because the
ambiguity rejection has several call sites and the v2 fast-path one
emits a byte-identical message: without the index the duplicate is
caught by the v2 writer, and without the subschema misclassification no
_rowaddr reaches the update stream. Either way the reconciler is never
involved and a message-only assertion would still pass.

The message is asserted instead of the Error variant on purpose. This
error crosses the sort/repartition in update_fragments, so DataFusion
returns it shared behind an Arc and the conversion in lance-core yields
Execution or InvalidInput depending on how many partitions still hold a
reference. The variant tracks the machine's CPU count; the text does
not.
@github-actions github-actions Bot added the chore label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

merge_insert violates a row-address ordering invariant during a full upsert

1 participant