tikv/tikv · critical

failed to get timestamp from PD

Error message

failed to get timestamp from PD

What it means

During server init, block_on(pd_client.get_tso()) requests a timestamp from Placement Driver to seed the concurrency manager; when PD is unreachable or returns an error the code panics with .expect("failed to get timestamp from PD"). Without an initial TSO the server cannot safely track max timestamps for transaction conflict detection, so startup aborts.

Source

Thrown at components/server/src/server2.rs:339

        let pd_client = TikvServerCore::connect_to_pd_cluster(
            &mut config,
            env.clone(),
            Arc::clone(&security_mgr),
        );

        // Initialize and check config
        let cfg_controller = TikvServerCore::init_config(config);
        let config = cfg_controller.get_current();

        let store_path = Path::new(&config.storage.data_dir).to_owned();

        let thread_count = config.server.background_thread_count;
        let background_worker = WorkerBuilder::new(BACKGROUND_WORKER_THREAD)
            .thread_count(thread_count)
            .create();

        // Initialize concurrency manager
        let latest_ts = block_on(pd_client.get_tso()).expect("failed to get timestamp from PD");
        let concurrency_manager = ConcurrencyManager::new_with_config(
            latest_ts,
            (config.storage.max_ts.cache_sync_interval * LIMIT_VALID_TIME_MULTIPLIER).into(),
            config
                .storage
                .max_ts
                .action_on_invalid_update
                .as_str()
                .try_into()
                .unwrap(),
            Some(pd_client.clone()),
            config.storage.max_ts.max_drift.0,
        );

        // use different quota for front-end and back-end requests
        let quota_limiter = Arc::new(QuotaLimiter::new(
            config.quota.foreground_cpu_time,
            config.quota.foreground_write_bandwidth,

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Verify PD endpoints, that PD is up and has a leader (pd-ctl / curl the PD health endpoint)
2. Fix network/firewall so TiKV can reach PD client port 2379
3. Match TLS certificates and CA between TiKV and PD if security is enabled
4. Restart TiKV after PD recovers; consider adding retry/backoff around get_tso

Example fix

// before
let latest_ts = block_on(pd_client.get_tso()).expect("failed to get timestamp from PD");
// after
let latest_ts = block_on(async {
    retry(pd_client.get_tso()).await
}).expect("failed to get timestamp from PD after retries");

// Before booting the server, verify PD reachability
for ep in &pd_endpoints {
    let health = format!("{}/health", ep);
    // curl/http GET health; require 200 with a PD leader set before proceeding
}

fn pd_healthy(health_json: &str) -> bool {
    health_json.contains("\"leader\"") && !health_json.contains("null")
}

let latest_ts = loop {
    match block_on(pd_client.get_tso()) {
        Ok(ts) => break ts,
        Err(e) => {
            warn!("get_tso failed, retrying"; "err" => %e);
            sleep(Duration::from_secs(1));
        }
    }
};

When it happens

Trigger: pd_client.get_tso() errors during node bootstrap: no PD endpoints reachable, TLS mismatch with PD, PD cluster has no leader, or network partition between TiKV and PD at startup.

Common situations: Wrong --pd-endpoints or PD down during cluster bring-up; firewall/security-group blocking the PD client port (2379); TLS config inconsistent between TiKV and PD; PD leader election in progress when TiKV starts.

Related errors

• failed to load all stores: {:?}
• failed to get timestamp from PD
• invalid max_ts update: {} exceeds the limit {}, source={}
• failed to load_latest_options {:?}
• fail to request PD {} err {:?}

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/c6ea8a40ec860349. Report an issue: GitHub.