tikv/tikv · error

wrong with global timer, cannot stepping.

Error message

wrong with global timer, cannot stepping.

What it means

This panic is the `.expect("wrong with global timer, cannot stepping.")` in `LeaderKeeper::elect_and_wait_all_ready` (components/snap_recovery/src/leader_keeper.rs:78). The leader-keeper loop steps force-leader campaigns for regions and then sleeps 10 seconds between rounds via `GLOBAL_TIMER_HANDLE.delay(...)` (a global tokio-based timer from tikv_util). The expect fires when the timer's delay future resolves to an error, meaning the global timer service is not running or has been shut down, so the loop cannot pace its retries.

Source

Thrown at components/snap_recovery/src/leader_keeper.rs:78

    pub fn new(router: Router, to_keep: impl IntoIterator<Item = u64>) -> Self {
        Self {
            router,

            not_leader: to_keep.into_iter().collect(),
            _ek: PhantomData,
        }
    }

    pub async fn elect_and_wait_all_ready(&mut self) {
        loop {
            let now = Instant::now();
            let res = self.step().await;
            info!("finished leader keeper stepping."; "result" => ?res, "take" => ?now.elapsed());
            GLOBAL_TIMER_HANDLE
                .delay(now + Duration::from_secs(10))
                .compat()
                .await
                .expect("wrong with global timer, cannot stepping.");
            if res.failed_leader.is_empty() {
                return;
            }
        }
    }

    pub async fn step(&mut self) -> StepResult {
        const CONCURRENCY: usize = 256;
        let r = Mutex::new(StepResult::default());
        let success = Mutex::new(HashSet::new());
        let regions = self.not_leader.iter().copied().collect::<Vec<_>>();
        for batch in regions.as_slice().chunks(CONCURRENCY) {
            let tasks = batch.iter().map(|region_id| async {
                match self.check_leader(*region_id).await {
                    Ok(_) => {
                        success.lock().unwrap().insert(*region_id);
                        return;
                    }

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Install the global timer before running recovery code: call `GLOBAL_TIMER_HANDLE.set(tikv_util::timer::build_global_timer(...))` (as tikv-server/BR entry points do) on your runtime.
2. Verify the process entry point matches upstream: run recovery through tikv-ctl/BR, which initializes the timer, rather than a hand-rolled main().
3. If embedding, replace the GLOBAL_TIMER_HANDLE delay with `tokio::time::sleep(Duration::from_secs(10))` in your fork.
4. Check for an earlier panic that shut down the timer thread and fix the root cause; restart the process.

Example fix

// before
GLOBAL_TIMER_HANDLE
    .delay(now + Duration::from_secs(10))
    .compat()
    .await
    .expect("wrong with global timer, cannot stepping.");

// after: degrade to a plain async sleep instead of panicking
if GLOBAL_TIMER_HANDLE
    .delay(now + Duration::from_secs(10))
    .compat()
    .await
    .is_err()
{
    error!("global timer unavailable, falling back to tokio sleep");
    tokio::time::sleep(Duration::from_secs(10)).await;
}

// Rust: confirm the global timer is installed before invoking leader keeper
// fn ensure_timer() -> Result<(), &'static str> {
//     if !GLOBAL_TIMER_HANDLE.is_set() { // helper equivalent: attempt a short delay
//         return Err("GLOBAL_TIMER_HANDLE not initialized; call GLOBAL_TIMER_HANDLE.set(build_global_timer(...)) on your runtime");
//     }
//     Ok(())
// }

// Rust narrowing: treat timer result as recoverable instead of panicking
match GLOBAL_TIMER_HANDLE.delay(now + Duration::from_secs(10)).compat().await {
    Ok(()) => { /* proceed */ }
    Err(e) => { error!("global timer failed"; "err" => ?e); /* fallback sleep */ }
}

// Wrap the whole leader-keeper campaign in error handling; never rely on the internal expect
async fn run_leader_keeper(mut keeper: LeaderKeeper<'_, EK, Router>) {
    loop {
        let res = keeper.step().await;
        if let Err(e) = GLOBAL_TIMER_HANDLE
            .delay(Instant::now() + Duration::from_secs(10))
            .compat()
            .await
        {
            error!("global timer unavailable"; "err" => ?e);
            tokio::time::sleep(Duration::from_secs(10)).await; // fallback pacing
        }
        if res.failed_leader.is_empty() { return; }
    }
}

When it happens

Trigger: Calling `elect_and_wait_all_ready()` from a runtime where the global timer was never installed: the recovery code runs on a tokio runtime created without `GLOBAL_TIMER_HANDLE.set()` / `build_global_timer`, or the timer thread was stopped before this call, or the future is polled outside the expected runtime context so the timer future yields an error.

Common situations: Embedding the snap_recovery LeaderKeeper in a custom tool (e.g. a BR-like binary or test harness) that builds its own tokio runtime and forgets to install the TiKV global timer; a runtime refactor that drops the `GLOBAL_TIMER_HANDLE.set(...)` bootstrap; tests driving the async fn on a plain `#[tokio::test]` runtime without the TiKV timer; timer thread crashed due to an earlier panic in the process.

Related errors

• invalid pd configuration: {:?}
• Number of threads can't be 0
• read meta file {} failed, error {:?}
• parse from bytes error {:?}
• unknown log format {}

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/50105981d0dc10ec. Report an issue: GitHub.