tikv/tikv · critical

invalid pd configuration: {:?}

Error message

invalid pd configuration: {:?}

What it means

read_queue::pop_front() panics with 'read_queue is empty but ready_cnt > 0' when the internal invariant between the ready_cnt counter and the reads VecDeque is broken: the counter says a ready read exists, but the deque is empty. This is a panic via expect(), not a recoverable error, and indicates a bookkeeping bug in Raft read-request tracking (ready/handled counters vs. queue contents), typically after a read was removed or completed twice.

Source

Thrown at cmd/tikv-ctl/src/main.rs:685

                    let from_key = from.map(|k| unescape(&k));
                    let to_key = to.map(|k| unescape(&k));
                    let bottommost = BottommostLevelCompaction::from(Some(bottommost.as_ref()));
                    if let Some(region) = region {
                        debug_executor
                            .compact_region(host, db_type, &cf, region, threads, bottommost);
                    } else {
                        debug_executor
                            .compact(host, db_type, &cf, from_key, to_key, threads, bottommost);
                    }
                }
                Cmd::Tombstone { regions, pd, force } => {
                    if let Some(pd_urls) = pd {
                        let cfg = PdConfig {
                            endpoints: pd_urls,
                            ..Default::default()
                        };
                        if let Err(e) = cfg.validate() {
                            panic!("invalid pd configuration: {:?}", e);
                        }
                        debug_executor.set_region_tombstone_after_remove_peer(mgr, &cfg, regions);
                    } else {
                        assert!(force);
                        debug_executor.set_region_tombstone_force(regions);
                    }
                }
                Cmd::RecoverMvcc {
                    read_only,
                    all,
                    threads,
                    regions,
                    pd: pd_urls,
                } => {
                    if all {
                        let threads = threads.unwrap();
                        if threads == 0 {
                            panic!("Number of threads can't be 0");

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Reproduce with panic backtrace and check which read id was double-handled; look for duplicate calls to complete/pop_front for the same ReadIndexRequest
2. Verify ready_cnt and handled_cnt are updated exactly once per read across all code paths that remove reads (including contexts removal)
3. Revert or review any local modifications to components/raftstore/src/store/read_queue.rs before filing upstream
4. Run raftstore unit tests (e.g. test read_queue) and nextest with EXTRA_CARGO_ARGS=read_queue to confirm invariant restoration
5. If reproducible on a released version, capture region/peer info and report upstream with the panic backtrace

Example fix

// before
self.ready_cnt -= 1;
self.handled_cnt += 1;
let mut res = self.reads.pop_front().expect("read_queue is empty but ready_cnt > 0");
// after (defensive guard for debugging)
assert!(self.ready_cnt > 0, "ready_cnt underflow");
self.ready_cnt -= 1;
self.handled_cnt += 1;
let mut res = match self.reads.pop_front() {
    Some(r) => r,
    None => panic!("read_queue desync: ready_cnt={} but queue empty", self.ready_cnt),
};

// Caller-side: never assume a ready read exists without tracking completion yourself
pub struct ReadQueueGuard<'a> {
    q: &'a mut ReadQueue<...>,
    completed: bool,
}
impl Drop for ReadQueueGuard<'_> {
    fn drop(&mut self) {
        if !self.completed {
            // ensure accounting stays in sync if a path forgot to complete
        }
    }
}
// Before popping: assert(queue.ready_cnt > 0 && queue.len() > 0) in debug builds

fn has_pending_read(q: &ReadQueue) -> bool {
    q.ready_cnt() > 0 && q.pending_len() > 0 && q.ready_cnt() <= q.pending_len()
}

When it happens

Trigger: Calling pop_front() when ready_cnt has been decremented/desynced from reads; double-completing the same ReadIndexRequest; a read removed from contexts/reads while its id was still counted as ready; memory corruption of the queue via duplicate apply of callbacks for the same ReadIndex task.

Common situations: Custom patches or backports to the raftstore ReadIndex path that alter pop_front/complete accounting; replica-read handling races in debug/fuzz builds; mixing TiKV versions of read_queue logic during upgrades.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• {} no match of persisted number {}, unpersisted readies: {:?
• {} should not applying snapshot.
• Number of threads can't be 0
• read meta file {} failed, error {:?}
• parse from bytes error {:?}

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/58db4c3242e505e5. Report an issue: GitHub.