Skip to content

fix(arrow): enforce exact Mosaic predicate filtering - #694

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:mosaic-exactness
Open

fix(arrow): enforce exact Mosaic predicate filtering#694
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:mosaic-exactness

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Mosaic only used row-group statistics for predicate pushdown, so core reads could return non-matching rows. Residual evaluation could also panic or fail open when predicates referenced physically missing columns.

Brief change log

  • Apply shared row-level residual filtering after Mosaic row-group pruning.
  • Include non-projected predicate columns in the scan.
  • Remap predicates against physical columns so missing columns follow NULL semantics.
  • Add regression coverage for missing columns, projection, deletion vectors, and direct core reads.

Tests

  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
  • cargo test --locked -p paimon --all-targets --features fulltext,vortex
  • cargo test --locked -p paimon-datafusion --test mosaic_tables

API and Format

Documentation

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant