Skip to content

fix(server): handle incomplete RocksDB with missing column families gracefully - #2404

Open
cauchy1988 wants to merge 1 commit into
apache:masterfrom
cauchy1988:fix/missing-cf-crash
Open

cauchy1988 wants to merge 1 commit into
apache:masterfrom
cauchy1988:fix/missing-cf-crash

Conversation

@cauchy1988

@cauchy1988 cauchy1988 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Problem

When a replica server crashes during DB::Open (e.g. OOM, kill -9), it may leave an incomplete rdb directory — the MANIFEST contains only partial column families (e.g. only pegasus_data without pegasus_meta). On restart, the server hits:

CHECK_PREFIX_MSG(!missing_meta_cf, "You must upgrade Pegasus server from 2.0");

and aborts immediately, making the replica unrecoverable.

Root Cause

  • The crash leaves a partially created RocksDB directory (rdb_path exists but is incomplete)
  • On restart, db_exist = true and the code enters the "open existing DB" path
  • check_column_families() detects missing_meta_cf = true or missing_data_cf = true
  • The CHECK_PREFIX_MSG causes a hard abort

An incomplete DB is unusable anyway (cannot read decree, data version, etc. from it), so crashing is unnecessary.

Fix

When missing_meta_cf || missing_data_cf is detected:

  1. Log a WARNING with details about the incomplete state
  2. Remove the corrupted rdb directory
  3. Set db_exist = false
  4. Fall through to create a fresh DB — the replica will re-sync data from the primary

The existing db_exist branch (LoadLatestOptions etc.) is wrapped in if (db_exist) to avoid operating on the deleted directory.

Testing

  • Verified that the fix resolves the crash in Docker development environment (Docker for Mac ARM64)
  • The replica successfully creates a new DB and re-syncs from primary after the corrupted directory is removed
  • Ran ./run.sh test -m base_api_test: integration_test.write_corrupt_db and integration_test.read_corrupt_db now pass. This also resolves base_api_test failed due to the failure of a replica server #2283, where the replica dropped on a corrupted DB (get_alive_replica_server_count() returned 2) and cascaded into 49 failures across the whole suite.

Related

Closes #2215
Closes #2283

…racefully

When a replica server crashes during DB::Open (e.g. OOM, kill -9), it
may leave an incomplete rdb directory with only partial column families
in the MANIFEST. On restart, the server hits CHECK_PREFIX_MSG and aborts
immediately, making the replica unrecoverable.

Instead of crashing, detect missing_meta_cf or missing_data_cf, remove
the corrupted rdb directory, set db_exist=false, and fall through to
create a fresh DB. The replica will re-sync data from the primary.

Closes apache#2215

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant