Skip to content

[core][mosaic] Preserve projected raw reads - #9086

Merged
JingsongLi merged 8 commits into
apache:masterfrom
jianguotian:fix/raw-file-split-read-projection
Aug 7, 2026
Merged

[core][mosaic] Preserve projected raw reads#9086
JingsongLi merged 8 commits into
apache:masterfrom
jianguotian:fix/raw-file-split-read-projection

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

RawFileSplitRead can be reused with different projected RowType values, but its cached FormatReaderMapping entries are keyed only by file schema and format. Reconfiguring the reader can therefore reuse a stale projection. In addition, lazily-created file readers need to retain the output type selected when each reader is created.

This PR:

  • invalidates cached format mappings only when withReadType receives a different RowType;
  • captures the current output type when a reader is created;
  • passes that same type through the mapping builder, lazy file-reader suppliers, and DataFileRecordReader;
  • preserves existing Manifest partition-column restoration when partition fields are requested.

The core change is format-neutral. It does not add Direct Bundle or writer behavior.

Tests

Added coverage for:

  • changed projections not sharing cached format mappings;
  • equal RowType instances reusing the existing mapping;
  • physical Mosaic files containing only business columns;
  • full, business-only, and partition-only projections;
  • row tracking and existing-versus-future lazy reader stability.

Focused validation:

mvn -B -ntp \
  -pl paimon-common,paimon-core,paimon-mosaic -am \
  -Dtest=ColumnarRowIteratorTest,RawFileSplitReadTest,PartitionColumnsOutsidePhysicalMosaicTest \
  -Dsurefire.failIfNoSpecifiedTests=false test
  • ColumnarRowIteratorTest: 1 passed
  • RawFileSplitReadTest: 2 passed
  • PartitionColumnsOutsidePhysicalMosaicTest: 2 passed
  • Checkstyle, Spotless, Maven Enforcer, and git diff --check: passed

@jianguotian jianguotian closed this Aug 7, 2026
@jianguotian jianguotian reopened this Aug 7, 2026

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit c43bd6c into apache:master Aug 7, 2026
17 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants