{"record":{"id":"b6d4e5939a8307a3","repo":"tikv/tikv","slug":"something-is-wrong-maybe-pd-do-not-ensure-all-tar","errorCode":null,"errorMessage":"something is wrong, maybe PD do not ensure all target peers exist before merging","messagePattern":"something is wrong, maybe PD do not ensure all target peers exist before merging","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"components/raftstore/src/store/fsm/store.rs","lineNumber":2699,"sourceCode":"            let (can_destroy, merge_to_this_peer) = maybe_destroy_source(\n                &meta,\n                region_id,\n                target.get_id(),\n                exist_region.get_id(),\n                msg.get_region_epoch().to_owned(),\n            );\n            if can_destroy {\n                if !merge_to_this_peer {\n                    regions_to_destroy.push(exist_region.get_id());\n                } else {\n                    error!(\n                        \"A new peer has a merge source peer\";\n                        \"region_id\" => region_id,\n                        \"peer_id\" => target.get_id(),\n                        \"source_region\" => ?exist_region,\n                    );\n                    if self.ctx.cfg.dev_assert {\n                        panic!(\n                            \"something is wrong, maybe PD do not ensure all target peers exist before merging\"\n                        );\n                    }\n                }\n                continue;\n            }\n            is_overlapped = true;\n            if msg.get_region_epoch().get_version() > exist_region.get_region_epoch().get_version()\n            {\n                // If new region's epoch version is greater than exist region's, the exist\n                // region may has been merged/splitted already.\n                let _ = self.ctx.router.force_send(\n                    exist_region.get_id(),\n                    PeerMsg::CasualMessage(Box::new(CasualMessage::RegionOverlapped)),\n                );\n            }\n        }\n","sourceCodeStart":2681,"sourceCodeEnd":2717,"githubUrl":"https://github.com/tikv/tikv/blob/78aedc1c81ef3f7d8bacc6e9d09f56460f134937/components/raftstore/src/store/fsm/store.rs#L2681-L2717","documentation":"During merge operations, when a new peer is created for a target region but the existing region data shows it was a merge-source peer, TiKV concludes PD did not ensure all target peers existed before the merge. By default it logs a warning and skips (`continue`), but with `dev_assert = true` in the raftstore config it panics. It signals a PD/raftstore contract violation around merge preconditions.","triggerScenarios":"`create_peer` observes a new peer whose history indicates it belonged to a merge source region, while PD's merge procedure was expected to have already created all target peers — i.e. PD issued a merge without pre-creating the target peer on this store, or a stale peer is being created late. Only panics when cfg.dev_assert is enabled.","commonSituations":"Dev/test clusters with dev_assert=true exercising region merges; PD scheduling misconfigurations or PD bugs issuing merge requests out of order; network partitions causing delayed peer creation during merge; TiKV/PD version skew with incompatible merge protocols.","solutions":["Disable `dev_assert` in the raftstore config for production clusters so this degrades to a warning","Verify PD logs/schedule to confirm merge preconditions (target peers exist on all stores) were met","Restart the store so peers are rebuilt from raft logs and PD heartbeats resync region state","Upgrade PD and TiKV to matched versions; if reproducible, report the merge scheduling sequence"],"exampleFix":"// before (tikv.toml)\n[raftstore]\ndev_assert = true\n// after\n[raftstore]\ndev_assert = false","handlingStrategy":"validation","validationCode":"// pre-merge check: confirm all target peers exist before scheduling merge\n// pd-ctl region <target_region_id>  # verify peers present on all stores\n// and in config:\n// [raftstore]\n// dev_assert = false","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep dev_assert=false in production","Verify PD merge scheduler preconditions and matched PD/TiKV versions","Watch PD logs for failed merge schedules and address root cause","Avoid network partitions during scheduled merges"],"tags":["raftstore","pd","region-merge"],"backgroundTag":"region-merge-precondition-violation","analyzedSha":"78aedc1c81ef3f7d8bacc6e9d09f56460f134937","analyzedAt":"2026-09-03T23:31:32.398Z","contentChangedAt":"2026-09-03T23:31:32.398Z","schemaVersion":2},"datasetVersion":"2026-09-11T07:07:21.782Z"}