{"record":{"id":"1cf1ea870b01f5b5","repo":"sgl-project/sglang","slug":"destination-mla-kv-descriptors-do-not-match-prefil","errorCode":null,"errorMessage":"Destination MLA KV descriptors do not match prefill pp configuration","messagePattern":"Destination MLA KV descriptors do not match prefill pp configuration","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/mori/conn.py","lineNumber":878,"sourceCode":"        if len(dst_mem_descs) < 2 or end_layer > dst_total_layers:\n            raise ValueError(\n                \"Destination KV descriptors do not match prefill pp configuration\"\n            )\n        dst_k_descs = dst_mem_descs[start_layer:end_layer]\n        dst_v_descs = dst_mem_descs[\n            dst_total_layers + start_layer : dst_total_layers + end_layer\n        ]\n        return src_k_descs, src_v_descs, dst_k_descs, dst_v_descs, num_local_layers\n\n    def _get_mla_mem_desc_slices(\n        self, dst_mem_descs: List[MemoryDesc]\n    ) -> tuple[List[MemoryDesc], List[MemoryDesc], int]:\n        src_descs = self.kv_mem_descs\n        num_local_layers = len(src_descs)\n        start_layer = self.kv_args.prefill_start_layer\n        end_layer = start_layer + num_local_layers\n        if end_layer > len(dst_mem_descs):\n            raise ValueError(\n                \"Destination MLA KV descriptors do not match prefill pp configuration\"\n            )\n        dst_slice = dst_mem_descs[start_layer:end_layer]\n        return src_descs, dst_slice, num_local_layers\n\n    def _submit_batch_transfer_plan(\n        self,\n        src_desc: MemoryDesc,\n        dst_desc: MemoryDesc,\n        plan: BatchTransferPlan,\n    ) -> List[TransferStatus]:\n        if plan.empty():\n            return []\n\n        transfer_uid = self.engine.allocate_transfer_uid()\n\n        statuses = self.engine.batch_write(\n            [src_desc],","sourceCodeStart":860,"sourceCodeEnd":896,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/mori/conn.py#L860-L896","documentation":"Raised when sending an MLA KV cache in a Mori prefill-decode disaggregation setup: the local prefill rank holds num_local_layers MLA KV descriptors, and prefill_start_layer + num_local_layers exceeds the number of descriptors registered by the destination decode instance. This means the layer-range arithmetic derived from the prefill PP (pipeline parallelism) configuration does not fit inside the decode side's descriptor table, i.e. the two processes disagree about layer counts or layer offsets.","triggerScenarios":"Calling send_kvcache (via _submit_kv_transfer) with kv_args.prefill_start_layer set such that prefill_start_layer + len(kv_mem_descs) > len(dst_mem_descs). Happens when PP sizes, layer partitioning, or model configuration differ between the prefill and decode engines.","commonSituations":"Launching prefill and decode instances with different --pp-size values, different model versions/layer counts, mismatched pipeline layer partition configs, or a stale peer registration (descriptor metadata exchanged at bootstrap does not match the actual engine topology).","solutions":["Verify prefill and decode servers use identical --pp-size and the same model (same number of layers)","Check that kv_args.prefill_start_layer for this PP rank plus its local layer count stays within the decode instance's total descriptor count","Re-register the KV descriptors (restart both engines) so the exchanged peer metadata matches current topology","If using custom layer partitioning, validate prefill_start_layer/prefill_end_layer config values against the model's layer count"],"exampleFix":"# before\npython -m sglang.launch_server --model ... --disaggregation-prefill --pp-size 2  # prefill\npython -m sglang.launch_server --model ... --disaggregation-decode --pp-size 1  # decode\n\n# after: match PP topology on both sides\npython -m sglang.launch_server --model ... --disaggregation-prefill --pp-size 2\npython -m sglang.launch_server --model ... --disaggregation-decode --pp-size 2","handlingStrategy":"validation","validationCode":"start = kv_args.prefill_start_layer\nlocal = len(conn.kv_mem_descs)\nassert start + local <= len(peer_info.dst_mem_descs), (\n    f\"layer range [{start}, {start + local}) exceeds decode descriptors \"\n    f\"({len(peer_info.dst_mem_descs)}) — check PP config parity\"\n)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always launch prefill and decode with the same --pp-size and model","Validate exchanged descriptor metadata at registration time rather than at transfer time","Add a startup assertion comparing layer counts across PD peers"],"tags":["disaggregation","prefill-decode","pipeline-parallel","mla","kv-cache"],"backgroundTag":"parallel-config-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}