tikv/tikv · critical

failed to load all stores: {:?}

Error message

failed to load all stores: {:?}

What it means

During RaftServer startup, load_all_stores fetches the full store list from Placement Driver via pd_client.get_all_stores(false) to initialize replication mode. If the PD RPC fails, startup cannot safely proceed (replication constraints are unknown), so the server panics instead of starting in an undefined replication state.

Source

Thrown at src/server/raft_server.rs:344

                api_version: self.api_version,
                ..ident
            };
            engines.kv.put_msg(keys::STORE_IDENT_KEY, &ident)?;
            engines.sync_kv()?;
        }
        Ok(())
    }

    fn alloc_id(&self) -> Result<u64> {
        let id = self.pd_client.alloc_id()?;
        Ok(id)
    }

    fn load_all_stores(&mut self, status: Option<ReplicationStatus>) {
        info!("initializing replication mode"; "status" => ?status, "store_id" => self.store.id);
        let stores = match self.pd_client.get_all_stores(false) {
            Ok(stores) => stores,
            Err(e) => panic!("failed to load all stores: {:?}", e),
        };
        let mut state = self.state.lock().unwrap();
        if let Some(s) = status {
            state.set_status(s);
        }
        for mut store in stores {
            state
                .group
                .register_store(store.id, store.take_labels().into());
        }
    }

    // Exported for tests.
    #[doc(hidden)]
    pub fn prepare_bootstrap_cluster(
        &self,
        engines: &Engines<EK, ER>,
        store_id: u64,

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Verify PD is reachable and has a leader: curl the PD /pd/api/v1/status or use pd-ctl.
2. Check --pd-endpoints configuration points at the correct PD cluster.
3. Check network connectivity/firewall and TLS certs between TiKV and PD.
4. Simply retry: this is often transient during PD startup — restart TiKV after PD is healthy.
5. Inspect pd_client logs for the underlying gRPC error code to narrow the cause.

Example fix

// before
# systemd starts tikv before pd is up, panic on boot
After=network.target
// after
After=pd.service  # or add retry loop / systemd-watchdog restart
Restart=on-failure

// Probe PD health before starting TiKV
for ep in pd_endpoints {
    if reqwest::get(format!("{}/pd/api/v1/status", ep)).is_ok() { break; }
}
// abort startup with a clear message if no PD responds

// Supervisor-level: restart TiKV with backoff when it exits due to PD unavailability
systemd unit:
  Restart=on-failure
  RestartSec=5s

When it happens

Trigger: pd_client.get_all_stores(false) returns Err during RaftServer::start; PD unreachable, PD leader election in progress, TLS/auth mismatch with PD, or PD returning an error response.

Common situations: TiKV started before PD cluster is available/elected a leader; wrong --pd-endpoints; network/firewall blocking PD port 2379; TLS certificate mismatch; PD cluster being restored or overloaded.

Related errors

• failed to get timestamp from PD
• fail to request PD {} err {:?}
• failed to load_latest_options {:?}
• invalid auto generated configuration file {}, err {}
• got new safe point {} which is less than current safe point

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/a8569ffbd5982fa5. Report an issue: GitHub.