{"record":{"id":"c6ea8a40ec860349","repo":"tikv/tikv","slug":"failed-to-get-timestamp-from-pd-c6ea8a","errorCode":null,"errorMessage":"failed to get timestamp from PD","messagePattern":"failed to get timestamp from PD","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"components/server/src/server2.rs","lineNumber":339,"sourceCode":"        let pd_client = TikvServerCore::connect_to_pd_cluster(\n            &mut config,\n            env.clone(),\n            Arc::clone(&security_mgr),\n        );\n\n        // Initialize and check config\n        let cfg_controller = TikvServerCore::init_config(config);\n        let config = cfg_controller.get_current();\n\n        let store_path = Path::new(&config.storage.data_dir).to_owned();\n\n        let thread_count = config.server.background_thread_count;\n        let background_worker = WorkerBuilder::new(BACKGROUND_WORKER_THREAD)\n            .thread_count(thread_count)\n            .create();\n\n        // Initialize concurrency manager\n        let latest_ts = block_on(pd_client.get_tso()).expect(\"failed to get timestamp from PD\");\n        let concurrency_manager = ConcurrencyManager::new_with_config(\n            latest_ts,\n            (config.storage.max_ts.cache_sync_interval * LIMIT_VALID_TIME_MULTIPLIER).into(),\n            config\n                .storage\n                .max_ts\n                .action_on_invalid_update\n                .as_str()\n                .try_into()\n                .unwrap(),\n            Some(pd_client.clone()),\n            config.storage.max_ts.max_drift.0,\n        );\n\n        // use different quota for front-end and back-end requests\n        let quota_limiter = Arc::new(QuotaLimiter::new(\n            config.quota.foreground_cpu_time,\n            config.quota.foreground_write_bandwidth,","sourceCodeStart":321,"sourceCodeEnd":357,"githubUrl":"https://github.com/tikv/tikv/blob/78aedc1c81ef3f7d8bacc6e9d09f56460f134937/components/server/src/server2.rs#L321-L357","documentation":"During server init, block_on(pd_client.get_tso()) requests a timestamp from Placement Driver to seed the concurrency manager; when PD is unreachable or returns an error the code panics with .expect(\"failed to get timestamp from PD\"). Without an initial TSO the server cannot safely track max timestamps for transaction conflict detection, so startup aborts.","triggerScenarios":"pd_client.get_tso() errors during node bootstrap: no PD endpoints reachable, TLS mismatch with PD, PD cluster has no leader, or network partition between TiKV and PD at startup.","commonSituations":"Wrong --pd-endpoints or PD down during cluster bring-up; firewall/security-group blocking the PD client port (2379); TLS config inconsistent between TiKV and PD; PD leader election in progress when TiKV starts.","solutions":["Verify PD endpoints, that PD is up and has a leader (pd-ctl / curl the PD health endpoint)","Fix network/firewall so TiKV can reach PD client port 2379","Match TLS certificates and CA between TiKV and PD if security is enabled","Restart TiKV after PD recovers; consider adding retry/backoff around get_tso"],"exampleFix":"// before\nlet latest_ts = block_on(pd_client.get_tso()).expect(\"failed to get timestamp from PD\");\n// after\nlet latest_ts = block_on(async {\n    retry(pd_client.get_tso()).await\n}).expect(\"failed to get timestamp from PD after retries\");","handlingStrategy":"retry","validationCode":"// Before booting the server, verify PD reachability\nfor ep in &pd_endpoints {\n    let health = format!(\"{}/health\", ep);\n    // curl/http GET health; require 200 with a PD leader set before proceeding\n}","typeGuard":"fn pd_healthy(health_json: &str) -> bool {\n    health_json.contains(\"\\\"leader\\\"\") && !health_json.contains(\"null\")\n}","tryCatchPattern":"let latest_ts = loop {\n    match block_on(pd_client.get_tso()) {\n        Ok(ts) => break ts,\n        Err(e) => {\n            warn!(\"get_tso failed, retrying\"; \"err\" => %e);\n            sleep(Duration::from_secs(1));\n        }\n    }\n};","preventionTips":["Verify PD endpoints and PD leader health before starting TiKV","Keep PD client port 2379 open in firewalls/security groups","Align TLS certs/CA between TiKV and PD","Add retry with backoff around initial TSO fetch at startup"],"tags":["rust","panic","pd","tso","network","startup"],"backgroundTag":"pd-unreachable","analyzedSha":"78aedc1c81ef3f7d8bacc6e9d09f56460f134937","analyzedAt":"2026-09-03T23:31:32.398Z","contentChangedAt":"2026-09-03T23:31:32.398Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}