{"record":{"id":"df9c42585c702fce","repo":"sgl-project/sglang","slug":"slice-size-exceeds-destination-token-capacity-for","errorCode":null,"errorMessage":"Slice size exceeds destination token capacity for TP slice transfer","messagePattern":"Slice size exceeds destination token capacity for TP slice transfer","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/mori/conn.py","lineNumber":954,"sourceCode":"        local_tp_rank = self.kv_args.engine_rank % prefill_tp_size\n        dst_tp_rank = peer_info.decode_tp_rank % decode_tp_size\n\n        if prefill_tp_size > decode_tp_size:\n            src_head_start = 0\n            num_heads_to_send = src_heads_per_rank\n            unique_head_idx = local_tp_rank // src_replication\n            dst_head_start = (unique_head_idx * src_heads_per_rank) % dst_heads_per_rank\n        else:\n            src_head_start = (dst_tp_rank * dst_heads_per_rank) % src_heads_per_rank\n            num_heads_to_send = dst_heads_per_rank\n            dst_head_start = 0\n\n        src_head_slice_offset = src_head_start * bytes_per_head_slice\n        dst_head_slice_offset = dst_head_start * bytes_per_head_slice\n        heads_bytes_per_token = num_heads_to_send * bytes_per_head_slice\n\n        if heads_bytes_per_token > bytes_per_token_dst:\n            raise ValueError(\n                \"Slice size exceeds destination token capacity for TP slice transfer\"\n            )\n\n        return TPSliceConfig(\n            page_size=page_size,\n            src_item_len=src_item_len,\n            dst_item_len=dst_item_len,\n            bytes_per_token_src=bytes_per_token_src,\n            bytes_per_token_dst=bytes_per_token_dst,\n            src_head_slice_offset=src_head_slice_offset,\n            dst_head_slice_offset=dst_head_slice_offset,\n            heads_bytes_per_token_to_send=heads_bytes_per_token,\n        )\n\n    def _build_tp_slice_transfer_plan(\n        self,\n        kv_indices: npt.NDArray[np.int32],\n        dst_indices: npt.NDArray[np.int32],","sourceCodeStart":936,"sourceCodeEnd":972,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/mori/conn.py#L936-L972","documentation":"During TP-sliced KV transfer, num_heads_to_send * bytes_per_head_slice exceeds bytes_per_token_dst — the slice this rank wants to write into a destination token is larger than the whole token slot on the decode side. The TP slicing math produced an overlapping or oversized head range for the destination layout.","triggerScenarios":"send_kvcache with heterogeneous TP (prefill_tp_size != decode_tp_size) where replication/head-range computation yields num_heads_to_send whose byte footprint exceeds the destination token capacity, e.g. mismatched total_kv_heads or head_dim between the two sides.","commonSituations":"Prefill and decode launched with different TP sizes on a model whose kv_heads do not divide evenly, mismatched head_dim/dtype causing different bytes_per_token_dst, or a model whose KV head layout the slice logic does not cover.","solutions":["Run prefill and decode with equal TP sizes (avoids TP-slice path entirely)","Verify both sides use the same model config (num_kv_heads, head_dim) and KV dtype","Check that total_kv_heads is divisible by both prefill_tp_size and decode_tp_size when using heterogeneous TP","Report/inspect TPSliceConfig inputs (num_heads_to_send, bytes_per_head_slice, bytes_per_token_dst) if configs look correct — may be a slicing bug for your head layout"],"exampleFix":"# before\n# prefill: --tp 8, decode: --tp 4 with a model with 8 kv_heads not dividing cleanly\n\n# after\n# use matching TP on both sides\npython -m sglang.launch_server ... --disaggregation-prefill --tp 8\npython -m sglang.launch_server ... --disaggregation-decode --tp 8","handlingStrategy":"validation","validationCode":"heads_bytes = num_heads_to_send * bytes_per_head_slice\nassert heads_bytes <= bytes_per_token_dst, (\n    f\"TP slice {heads_bytes}B exceeds destination token {bytes_per_token_dst}B\"\n)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer equal TP sizes across prefill/decode to bypass the slice path","Verify num_kv_heads divides both TP sizes before enabling heterogeneous TP","Log TPSliceConfig inputs once at bootstrap as a canary"],"tags":["disaggregation","tensor-parallel","heterogeneous-tp","kv-cache"],"backgroundTag":"parallel-config-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}