tikv/tikv · critical

[peer = {}] region {:?} unexpected pending states {:?}

Error message

[peer = {}] region {:?} unexpected pending states {:?}

What it means

PeerStorage::clear_meta_in_kv_and_raft removes a peer's metadata (apply state, raft state, region state) from the KV engine during peer destroy / peer-meta clear. It checks the pending destroy/raft-cluster status registered for the region; if the status is not one of the expected variants (mismatched peer id or unexpected pending state), the cleanup aborts with this panic because proceeding could leave partial metadata or delete the wrong peer's data.

Source

Thrown at components/raftstore/src/store/peer_storage.rs:1207

        if initialized {
            assert_eq!(pending.get(&region.get_id()), None);
            (None, true)
        } else if let Some(status) = pending.get(&region.get_id()) {
            if *status == (peer_id, false) {
                pending.remove(&region.get_id());
                // Hold the lock to avoid apply worker applies split.
                (Some(pending), true)
            } else if *status == (peer_id, true) {
                // It's already marked to split by apply worker, skip delete.
                (None, false)
            } else {
                // Peer id can't be different as router should exist all the time, their is no
                // chance for store to insert a different peer id. And apply worker should skip
                // split when meeting a different id.
                let status = *status;
                // Avoid panic with lock.
                drop(pending);
                panic!(
                    "[peer = {}] region {:?} unexpected pending states {:?}",
                    peer_id, region, status
                );
            }
        } else {
            // The status is inserted when it's created. It will be removed in following
            // cases:
            // - By apply worker as it fails to split due to region state key. This is
            //   impossible to reach this code path because the delete write batch is not
            //   persisted yet.
            // - By store fsm as it fails to create peer, which is also invalid obviously.
            // - By peer fsm after persisting snapshot, then it should be initialized.
            // - By peer fsm after split.
            // - By peer fsm when destroy, which should go the above branch instead.
            (None, false)
        }
    } else {
        (None, true)

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Upgrade to the latest patch release of your TiKV version — several destroy/merge races were fixed over time.
2. From the panic log capture peer_id, region, and status; verify with tikv-ctl whether the region actually exists on this store (stale peer) and use `tikv-ctl remove-peer` / unsafe recovery if metadata is inconsistent.
3. Check for a crash mid-destroy (tombstone vs pending status) and confirm all replicas of the region agree on the peer id via PD.
4. Restart the store and let raft rebuild the region if the pending status is stale; do not hand-edit KV metadata without tikv-ctl.
5. If reproducible in a dev environment, add logging around pending destroy status insertion / check whether the apply worker skipped a split with a different peer id, and file a raftstore issue.

// The panic aborts the TiKV process, so handle it at deployment level
// (systemd/k8s restart) and remediate metadata with tikv-ctl:
//   tikv-ctl --db /path/to/db raft region <id> --check
// then, if the peer is stale/tombstoned:
//   tikv-ctl remove-peer --region <id> --peer <peer_id>
// Never try to catch this in-process: it is a fail-fast invariant.

When it happens

Trigger: Destroying a peer (region merge, conf-change removal, unsafe recovery) or handling clear-peer-meta when the pending status inserted at peer creation does not match the peer id / expected state — e.g. a different peer id is registered for the region, the status entry was already consumed, or a split/merge raced with destroy.

Common situations: Region merge or conf-change storms racing with store restart, apply worker observing a different peer id after region re-creation, corrupted or stale pending-destroy status left in the engine after a crash, or mixed-version rolling upgrades changing the status enum layout.

Related errors

• must wait ready first to start destroying
• task has a different peer id, region {} peer_id {} != prev_p
• {} prepare to destroy err {:?}
• invalid pd configuration: {:?}
• Number of threads can't be 0

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/ac32bdb765ad2d55. Report an issue: GitHub.