{"record":{"id":"58db4c3242e505e5","repo":"tikv/tikv","slug":"invalid-pd-configuration","errorCode":null,"errorMessage":"invalid pd configuration: {:?}","messagePattern":"invalid pd configuration: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/tikv-ctl/src/main.rs","lineNumber":685,"sourceCode":"                    let from_key = from.map(|k| unescape(&k));\n                    let to_key = to.map(|k| unescape(&k));\n                    let bottommost = BottommostLevelCompaction::from(Some(bottommost.as_ref()));\n                    if let Some(region) = region {\n                        debug_executor\n                            .compact_region(host, db_type, &cf, region, threads, bottommost);\n                    } else {\n                        debug_executor\n                            .compact(host, db_type, &cf, from_key, to_key, threads, bottommost);\n                    }\n                }\n                Cmd::Tombstone { regions, pd, force } => {\n                    if let Some(pd_urls) = pd {\n                        let cfg = PdConfig {\n                            endpoints: pd_urls,\n                            ..Default::default()\n                        };\n                        if let Err(e) = cfg.validate() {\n                            panic!(\"invalid pd configuration: {:?}\", e);\n                        }\n                        debug_executor.set_region_tombstone_after_remove_peer(mgr, &cfg, regions);\n                    } else {\n                        assert!(force);\n                        debug_executor.set_region_tombstone_force(regions);\n                    }\n                }\n                Cmd::RecoverMvcc {\n                    read_only,\n                    all,\n                    threads,\n                    regions,\n                    pd: pd_urls,\n                } => {\n                    if all {\n                        let threads = threads.unwrap();\n                        if threads == 0 {\n                            panic!(\"Number of threads can't be 0\");","sourceCodeStart":667,"sourceCodeEnd":703,"githubUrl":"https://github.com/tikv/tikv/blob/78aedc1c81ef3f7d8bacc6e9d09f56460f134937/cmd/tikv-ctl/src/main.rs#L667-L703","documentation":"read_queue::pop_front() panics with 'read_queue is empty but ready_cnt > 0' when the internal invariant between the ready_cnt counter and the reads VecDeque is broken: the counter says a ready read exists, but the deque is empty. This is a panic via expect(), not a recoverable error, and indicates a bookkeeping bug in Raft read-request tracking (ready/handled counters vs. queue contents), typically after a read was removed or completed twice.","triggerScenarios":"Calling pop_front() when ready_cnt has been decremented/desynced from reads; double-completing the same ReadIndexRequest; a read removed from contexts/reads while its id was still counted as ready; memory corruption of the queue via duplicate apply of callbacks for the same ReadIndex task.","commonSituations":"Custom patches or backports to the raftstore ReadIndex path that alter pop_front/complete accounting; replica-read handling races in debug/fuzz builds; mixing TiKV versions of read_queue logic during upgrades.","solutions":["Reproduce with panic backtrace and check which read id was double-handled; look for duplicate calls to complete/pop_front for the same ReadIndexRequest","Verify ready_cnt and handled_cnt are updated exactly once per read across all code paths that remove reads (including contexts removal)","Revert or review any local modifications to components/raftstore/src/store/read_queue.rs before filing upstream","Run raftstore unit tests (e.g. test read_queue) and nextest with EXTRA_CARGO_ARGS=read_queue to confirm invariant restoration","If reproducible on a released version, capture region/peer info and report upstream with the panic backtrace"],"exampleFix":"// before\nself.ready_cnt -= 1;\nself.handled_cnt += 1;\nlet mut res = self.reads.pop_front().expect(\"read_queue is empty but ready_cnt > 0\");\n// after (defensive guard for debugging)\nassert!(self.ready_cnt > 0, \"ready_cnt underflow\");\nself.ready_cnt -= 1;\nself.handled_cnt += 1;\nlet mut res = match self.reads.pop_front() {\n    Some(r) => r,\n    None => panic!(\"read_queue desync: ready_cnt={} but queue empty\", self.ready_cnt),\n};","handlingStrategy":"validation","validationCode":"// Caller-side: never assume a ready read exists without tracking completion yourself\npub struct ReadQueueGuard<'a> {\n    q: &'a mut ReadQueue<...>,\n    completed: bool,\n}\nimpl Drop for ReadQueueGuard<'_> {\n    fn drop(&mut self) {\n        if !self.completed {\n            // ensure accounting stays in sync if a path forgot to complete\n        }\n    }\n}\n// Before popping: assert(queue.ready_cnt > 0 && queue.len() > 0) in debug builds","typeGuard":"fn has_pending_read(q: &ReadQueue) -> bool {\n    q.ready_cnt() > 0 && q.pending_len() > 0 && q.ready_cnt() <= q.pending_len()\n}","tryCatchPattern":null,"preventionTips":["Complete each ReadIndexRequest exactly once and centralize ready_cnt/handled_cnt updates in one place","Run raftstore read_queue unit tests and debug assertions in CI","Review any backport/patch touching read_queue.rs for counter desync","Enable debug_assert invariant checks (ready_cnt <= reads.len()) during development"],"tags":["raftstore","read-index","invariant-violation","panic"],"backgroundTag":"internal-invariant-violation","analyzedSha":"78aedc1c81ef3f7d8bacc6e9d09f56460f134937","analyzedAt":"2026-09-03T23:31:32.398Z","contentChangedAt":"2026-09-03T23:31:32.398Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}