Skip to content

[flink][spark] Support manifest sort in compact_manifest - #9083

Open
discivigour wants to merge 5 commits into
apache:masterfrom
discivigour:j/manifestSortAction
Open

[flink][spark] Support manifest sort in compact_manifest#9083
discivigour wants to merge 5 commits into
apache:masterfrom
discivigour:j/manifestSortAction

Conversation

@discivigour

@discivigour discivigour commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

Extend the compact_manifest procedure so manifest sorting can be enabled and tuned for a single invocation without altering persistent table options.

This change:

  • adds optional manifest_sort_enabled, manifest_sort_partition_field, and manifest_sort_max_rewrite_size arguments to the Flink and Spark procedures;
  • keeps table options when dedicated arguments are omitted, lets options override table values, and gives dedicated arguments the highest precedence;
  • adds a Flink compact_manifest action with corresponding command-line options;
  • reports the manifest file count in each sorted-run level during dry run when manifest sorting is enabled; and
  • preserves no-op behavior when there is no manifest rewrite to commit.

Tests

  • mvn -pl paimon-core,paimon-flink/paimon-flink-common -Pflink1 -DskipTests -Dmaven.repo.local=/opt/homebrew/opt/maven/repository spotless:check
  • mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -Pflink1 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=CompactManifestProcedureITCase -Dmaven.repo.local=/opt/homebrew/opt/maven/repository test
  • mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark3 -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.procedure.CompactManifestProcedureTest -Dtest=none -Dmaven.repo.local=/opt/homebrew/opt/maven/repository test
  • git diff --check origin/master...HEAD

@discivigour
discivigour marked this pull request as ready for review August 7, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant