tikv/tikv · error

something is wrong, maybe PD do not ensure all target peers

Error message

something is wrong, maybe PD do not ensure all target peers exist before merging

What it means

During merge operations, when a new peer is created for a target region but the existing region data shows it was a merge-source peer, TiKV concludes PD did not ensure all target peers existed before the merge. By default it logs a warning and skips (`continue`), but with `dev_assert = true` in the raftstore config it panics. It signals a PD/raftstore contract violation around merge preconditions.

Source

Thrown at components/raftstore/src/store/fsm/store.rs:2699

            let (can_destroy, merge_to_this_peer) = maybe_destroy_source(
                &meta,
                region_id,
                target.get_id(),
                exist_region.get_id(),
                msg.get_region_epoch().to_owned(),
            );
            if can_destroy {
                if !merge_to_this_peer {
                    regions_to_destroy.push(exist_region.get_id());
                } else {
                    error!(
                        "A new peer has a merge source peer";
                        "region_id" => region_id,
                        "peer_id" => target.get_id(),
                        "source_region" => ?exist_region,
                    );
                    if self.ctx.cfg.dev_assert {
                        panic!(
                            "something is wrong, maybe PD do not ensure all target peers exist before merging"
                        );
                    }
                }
                continue;
            }
            is_overlapped = true;
            if msg.get_region_epoch().get_version() > exist_region.get_region_epoch().get_version()
            {
                // If new region's epoch version is greater than exist region's, the exist
                // region may has been merged/splitted already.
                let _ = self.ctx.router.force_send(
                    exist_region.get_id(),
                    PeerMsg::CasualMessage(Box::new(CasualMessage::RegionOverlapped)),
                );
            }
        }

View on GitHub (pinned to 78aedc1c81)

Solutions

1. Disable `dev_assert` in the raftstore config for production clusters so this degrades to a warning
2. Verify PD logs/schedule to confirm merge preconditions (target peers exist on all stores) were met
3. Restart the store so peers are rebuilt from raft logs and PD heartbeats resync region state
4. Upgrade PD and TiKV to matched versions; if reproducible, report the merge scheduling sequence

Example fix

// before (tikv.toml)
[raftstore]
dev_assert = true
// after
[raftstore]
dev_assert = false

// pre-merge check: confirm all target peers exist before scheduling merge
// pd-ctl region <target_region_id>  # verify peers present on all stores
// and in config:
// [raftstore]
// dev_assert = false

When it happens

Trigger: `create_peer` observes a new peer whose history indicates it belonged to a merge source region, while PD's merge procedure was expected to have already created all target peers — i.e. PD issued a merge without pre-creating the target peer on this store, or a stale peer is being created late. Only panics when cfg.dev_assert is enabled.

Common situations: Dev/test clusters with dev_assert=true exercising region merges; PD scheduling misconfigurations or PD bugs issuing merge requests out of order; network partitions causing delayed peer creation during merge; TiKV/PD version skew with incompatible merge protocols.

Related errors

• expect empty prepare merge status {}: {:?}
• expect WaitForTrimStatus {}: {:?}
• {} rollbacks a wrong merge: {} != {}
• {} unexpected merge result: {:?} {:?} {:?}
• {} is applying atomic snapshot on getting merge result, targ

AI-assisted analysis of tikv/tikv@78aedc1c81 (2026-09-03). Data as JSON: /api/errors/b6d4e5939a8307a3. Report an issue: GitHub.