Skip to content

[core] Fix stale file index cleanup - #9085

Open
huangxiaopingRD wants to merge 5 commits into
apache:masterfrom
huangxiaopingRD:fix-index-clean
Open

[core] Fix stale file index cleanup#9085
huangxiaopingRD wants to merge 5 commits into
apache:masterfrom
huangxiaopingRD:fix-index-clean

Conversation

@huangxiaopingRD

Copy link
Copy Markdown
Contributor

Purpose

Fix rewrite_file_index retaining stale index types when a column switches from one File Index type to another. Track configured index types per column and remove obsolete types during rewrite. Add regression coverage for Bloom Filter to Bitmap replacement.

Tests

Added regression tests for replacing stale File Index types during rewrite_file_index.

@JingsongLi

Copy link
Copy Markdown
Contributor
  • Both methods of calling the Bloom → Bitmap use case of the local RewriteFileIndexProcedureITCase failed.

  • When the external index becomes a smaller embedded index, dataFileMeta.copy(byte[]) retains the old extraFiles, causing the old Bloom file to remain in the manifest and be read; the reverse switch may also retain the old embedded index.

  • Both external and embedded storage fields need to be updated and cleaned up simultaneously. There are currently 10 failures in GitHub CI.

@huangxiaopingRD

huangxiaopingRD commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author
  • Both methods of calling the Bloom → Bitmap use case of the local RewriteFileIndexProcedureITCase failed.
  • When the external index becomes a smaller embedded index, dataFileMeta.copy(byte[]) retains the old extraFiles, causing the old Bloom file to remain in the manifest and be read; the reverse switch may also retain the old embedded index.
  • Both external and embedded storage fields need to be updated and cleaned up simultaneously. There are currently 10 failures in GitHub CI.

@JingsongLi Thanks for your review. I have fixed the issue. Please review again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants