PHOENIX-7988 Idle co-active RegionServer stays wedged in STORE_AND_FORWARD, pinning the group at ACTIVE_NOT_IN_SYNC - #2600
Open
tkhurana wants to merge 1 commit into
Conversation
…RWARD, pinning the group at ACTIVE_NOT_IN_SYNC A co-active RS in STORE_AND_FORWARD promotes back to SYNC_AND_FORWARD via processNoMoreRoundsLeft only when the caught-up guard passes. That guard scanned the next round's shard, which is a shared directory holding every co-active RS's live OPENFORWRITE rotation writer, so an idle or out-claimed RS never saw it empty and stayed pinned in STORE_AND_FORWARD, wedging the group at ACTIVE_NOT_IN_SYNC indefinitely. Split the guard: the RS's own mode flip is gated only on an empty in-progress directory (the per-RS forward-health signal), while the shared in-sync status claim keeps the full caught-up guard. The mode flip is self-validating since SyncAndForwardModeImpl.onEnter must reach the peer, so an optimistic promotion against a dead peer bounces back to STORE_AND_FORWARD. Adds a VisibleForTesting tracker-injecting constructor and three focused tests covering the wedge fix, the peer-down backlog case (no promotion), and the fully-caught-up case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A co-active RS in STORE_AND_FORWARD promotes back to SYNC_AND_FORWARD via processNoMoreRoundsLeft only when the caught-up guard passes. That guard scanned the next round's shard, which is a shared directory holding every co-active RS's live OPENFORWRITE rotation writer, so an idle or out-claimed RS never saw it empty and stayed pinned in STORE_AND_FORWARD, wedging the group at ACTIVE_NOT_IN_SYNC indefinitely.
Split the guard: the RS's own mode flip is gated only on an empty in-progress directory (the per-RS forward-health signal), while the shared in-sync status claim keeps the full caught-up guard. The mode flip is self-validating since SyncAndForwardModeImpl.onEnter must reach the peer, so an optimistic promotion against a dead peer bounces back to STORE_AND_FORWARD.
Adds a VisibleForTesting tracker-injecting constructor and three focused tests covering the wedge fix, the peer-down backlog case (no promotion), and the fully-caught-up case.