tikv/tikv · critical

failed to open raftdb for migration

Error message

failed to open raftdb for migration

What it means

In the raftdb-to-raft-engine migration path, the server opens the existing raftdb (RocksDB) to dump its data into the raft engine, panicking if raftdb cannot be opened. The migration cannot proceed without reading the source data, so failure is fatal.

Source

Thrown at components/server/src/common.rs:848

        );
        let should_dump = raft_data_state_machine.before_open_target();

        let raft_config = config.raft_engine.config();
        let raft_engine =
            RaftLogEngine::new(raft_config, key_manager.clone(), get_io_rate_limiter())
                .expect("failed to open raft engine");

        if should_dump {
            let config_raftdb = &config.raftdb;
            let raft_db_opts = config_raftdb.build_opt(env.clone(), None);
            let raft_cf_opts = config_raftdb.build_cf_opts(block_cache);
            let raftdb = engine_rocks::util::new_engine_opt_with_snapshot_sequence_number_check(
                &config.raft_store.raftdb_path,
                raft_db_opts,
                raft_cf_opts,
                config_raftdb.enable_snapshot_sequence_number_check,
            )
            .expect("failed to open raftdb for migration");
            dump_raftdb_to_raft_engine(&raftdb, &raft_engine, 8 /* threads */);
            raftdb.stop();
            drop(raftdb);
            raft_data_state_machine.after_dump_data();
        }
        (raft_engine, None)
    }
}

const DEFAULT_ENGINE_METRICS_RESET_INTERVAL: Duration = Duration::from_millis(60_000);
pub struct EngineMetricsManager<EK: KvEngine, ER: RaftEngine> {
    tablet_registry: TabletRegistry<EK>,
    kv_statistics: Option<Arc<RocksStatistics>>,
    in_memory_engine_statistics: Option<Arc<InMemoryEngineStatistics>>,
    kv_is_titan: bool,
    raft_engine: ER,
    raft_statistics: Option<Arc<RocksStatistics>>,
    last_reset: Instant,

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Read the RocksDB error logged above the panic to identify lock vs corruption vs IO
2. Ensure the raftdb dir is not held by another process and permissions/disk are fine
3. Repair via tikv-ctl or restore from backup before retrying the migration

Example fix

// before (panic inside build)
.expect("failed to open raftdb for migration")
// after (operator pre-check)
tikv-ctl raft log --db /data/tikv/raft  # validate raftdb before restarting with raft-engine

// verify raftdb opens read-only before enabling raft-engine migration
let opts = ROpts::default(); // open with error_if_corrupt to validate
tikv_ctl_check_raftdb(&config.raft_store.raftdb_path);

When it happens

Trigger: ServerBuilder::build with raftdb as current storage and should_dump true, when new_engine_opt_with_snapshot_sequence_number_check fails on config.raft_store.raftdb_path — corrupt RocksDB data, stale LOCK, bad options, or decryption failures.

Common situations: Upgrading a cluster whose raftdb directory was damaged by an earlier crash; running migration while another process holds the data dir; switching from unencrypted to encrypted storage without proper key setup.

Related errors

• failed to open raftdb
• failed to load_latest_options {:?}
• failed to open raft engine for migration
• rocksdb background error. db: {}, reason: {}, error: {}
• {:?}

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/121dacd8755bf588. Report an issue: GitHub.