{"record":{"id":"a8569ffbd5982fa5","repo":"tikv/tikv","slug":"failed-to-load-all-stores","errorCode":null,"errorMessage":"failed to load all stores: {:?}","messagePattern":"failed to load all stores: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"src/server/raft_server.rs","lineNumber":344,"sourceCode":"                api_version: self.api_version,\n                ..ident\n            };\n            engines.kv.put_msg(keys::STORE_IDENT_KEY, &ident)?;\n            engines.sync_kv()?;\n        }\n        Ok(())\n    }\n\n    fn alloc_id(&self) -> Result<u64> {\n        let id = self.pd_client.alloc_id()?;\n        Ok(id)\n    }\n\n    fn load_all_stores(&mut self, status: Option<ReplicationStatus>) {\n        info!(\"initializing replication mode\"; \"status\" => ?status, \"store_id\" => self.store.id);\n        let stores = match self.pd_client.get_all_stores(false) {\n            Ok(stores) => stores,\n            Err(e) => panic!(\"failed to load all stores: {:?}\", e),\n        };\n        let mut state = self.state.lock().unwrap();\n        if let Some(s) = status {\n            state.set_status(s);\n        }\n        for mut store in stores {\n            state\n                .group\n                .register_store(store.id, store.take_labels().into());\n        }\n    }\n\n    // Exported for tests.\n    #[doc(hidden)]\n    pub fn prepare_bootstrap_cluster(\n        &self,\n        engines: &Engines<EK, ER>,\n        store_id: u64,","sourceCodeStart":326,"sourceCodeEnd":362,"githubUrl":"https://github.com/tikv/tikv/blob/78aedc1c81ef3f7d8bacc6e9d09f56460f134937/src/server/raft_server.rs#L326-L362","documentation":"During RaftServer startup, load_all_stores fetches the full store list from Placement Driver via pd_client.get_all_stores(false) to initialize replication mode. If the PD RPC fails, startup cannot safely proceed (replication constraints are unknown), so the server panics instead of starting in an undefined replication state.","triggerScenarios":"pd_client.get_all_stores(false) returns Err during RaftServer::start; PD unreachable, PD leader election in progress, TLS/auth mismatch with PD, or PD returning an error response.","commonSituations":"TiKV started before PD cluster is available/elected a leader; wrong --pd-endpoints; network/firewall blocking PD port 2379; TLS certificate mismatch; PD cluster being restored or overloaded.","solutions":["Verify PD is reachable and has a leader: curl the PD /pd/api/v1/status or use pd-ctl.","Check --pd-endpoints configuration points at the correct PD cluster.","Check network connectivity/firewall and TLS certs between TiKV and PD.","Simply retry: this is often transient during PD startup — restart TiKV after PD is healthy.","Inspect pd_client logs for the underlying gRPC error code to narrow the cause."],"exampleFix":"// before\n# systemd starts tikv before pd is up, panic on boot\nAfter=network.target\n// after\nAfter=pd.service  # or add retry loop / systemd-watchdog restart\nRestart=on-failure","handlingStrategy":"retry","validationCode":"// Probe PD health before starting TiKV\nfor ep in pd_endpoints {\n    if reqwest::get(format!(\"{}/pd/api/v1/status\", ep)).is_ok() { break; }\n}\n// abort startup with a clear message if no PD responds","typeGuard":null,"tryCatchPattern":"// Supervisor-level: restart TiKV with backoff when it exits due to PD unavailability\nsystemd unit:\n  Restart=on-failure\n  RestartSec=5s","preventionTips":["Ensure PD is started and has elected a leader before TiKV","Configure correct --pd-endpoints and validate connectivity (port 2379) and TLS certs","Use a process supervisor with automatic restart so transient PD outages are retried","Monitor PD reachability from TiKV hosts"],"tags":["rust","panic","pd","startup","network","grpc"],"backgroundTag":"pd-unreachable","analyzedSha":"78aedc1c81ef3f7d8bacc6e9d09f56460f134937","analyzedAt":"2026-09-03T23:31:32.398Z","contentChangedAt":"2026-09-03T23:31:32.398Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}