Skip to content

Load: Fix active load reliability issues#18224

Merged
jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:fix/load-active-reliability
Jul 21, 2026
Merged

Load: Fix active load reliability issues#18224
jt2594838 merged 3 commits into
apache:masterfrom
Caideyipi:fix/load-active-reliability

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Active load handoff

  • Transfer each TsFile and its resource/mods files through a unique directory, publish the TsFile last, and clean partial output on failure.
  • Keep source files until the whole handoff succeeds and isolate same-name file groups from one another.

Scanner and loader lifecycle

  • Deduplicate scanner candidates derived from TsFiles and companion files.
  • Prevent internal attribute/transfer directories from being inferred as table databases.
  • Clear stale pending state across loader restarts and clean empty transfer directories after processing.

Load resource cleanup

  • Release cached LOAD memory when split/routing exits early.
  • Retry deletion of the TsFile and each companion independently so one failure does not skip the remaining cleanup.

Test coverage

  • Datanode load unit-test suite: 41 tests passed.
  • Final focused regression suite: 15 tests passed.
  • Spotless check, Checkstyle, and git diff --check passed.

This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.

Key changed/added classes (or packages if there are too many classes) in this PR
  • LoadSingleTsFileNode and LoadTsFileScheduler
  • ActiveLoadDirScanner, ActiveLoadPendingQueue, and ActiveLoadTsFileLoader
  • LoadUtil

@jt2594838
jt2594838 merged commit ea0b009 into apache:master Jul 21, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants