sgl-project/sglang · critical · ValueError

Slice size exceeds destination token capacity for TP slice t

Error message

Slice size exceeds destination token capacity for TP slice transfer

What it means

During TP-sliced KV transfer, num_heads_to_send * bytes_per_head_slice exceeds bytes_per_token_dst — the slice this rank wants to write into a destination token is larger than the whole token slot on the decode side. The TP slicing math produced an overlapping or oversized head range for the destination layout.

Source

Thrown at python/sglang/srt/disaggregation/mori/conn.py:954

        local_tp_rank = self.kv_args.engine_rank % prefill_tp_size
        dst_tp_rank = peer_info.decode_tp_rank % decode_tp_size

        if prefill_tp_size > decode_tp_size:
            src_head_start = 0
            num_heads_to_send = src_heads_per_rank
            unique_head_idx = local_tp_rank // src_replication
            dst_head_start = (unique_head_idx * src_heads_per_rank) % dst_heads_per_rank
        else:
            src_head_start = (dst_tp_rank * dst_heads_per_rank) % src_heads_per_rank
            num_heads_to_send = dst_heads_per_rank
            dst_head_start = 0

        src_head_slice_offset = src_head_start * bytes_per_head_slice
        dst_head_slice_offset = dst_head_start * bytes_per_head_slice
        heads_bytes_per_token = num_heads_to_send * bytes_per_head_slice

        if heads_bytes_per_token > bytes_per_token_dst:
            raise ValueError(
                "Slice size exceeds destination token capacity for TP slice transfer"
            )

        return TPSliceConfig(
            page_size=page_size,
            src_item_len=src_item_len,
            dst_item_len=dst_item_len,
            bytes_per_token_src=bytes_per_token_src,
            bytes_per_token_dst=bytes_per_token_dst,
            src_head_slice_offset=src_head_slice_offset,
            dst_head_slice_offset=dst_head_slice_offset,
            heads_bytes_per_token_to_send=heads_bytes_per_token,
        )

    def _build_tp_slice_transfer_plan(
        self,
        kv_indices: npt.NDArray[np.int32],
        dst_indices: npt.NDArray[np.int32],

View on GitHub (pinned to 0132848349)

Solutions

Run prefill and decode with equal TP sizes (avoids TP-slice path entirely)
Verify both sides use the same model config (num_kv_heads, head_dim) and KV dtype
Check that total_kv_heads is divisible by both prefill_tp_size and decode_tp_size when using heterogeneous TP
Report/inspect TPSliceConfig inputs (num_heads_to_send, bytes_per_head_slice, bytes_per_token_dst) if configs look correct — may be a slicing bug for your head layout

Example fix

# before
# prefill: --tp 8, decode: --tp 4 with a model with 8 kv_heads not dividing cleanly

# after
# use matching TP on both sides
python -m sglang.launch_server ... --disaggregation-prefill --tp 8
python -m sglang.launch_server ... --disaggregation-decode --tp 8

Defensive patterns

Strategy: validation

Validate before calling

heads_bytes = num_heads_to_send * bytes_per_head_slice
assert heads_bytes <= bytes_per_token_dst, (
    f"TP slice {heads_bytes}B exceeds destination token {bytes_per_token_dst}B"
)

Prevention

Prefer equal TP sizes across prefill/decode to bypass the slice path
Verify num_kv_heads divides both TP sizes before enabling heterogeneous TP
Log TPSliceConfig inputs once at bootstrap as a canary

When it happens

Trigger: send_kvcache with heterogeneous TP (prefill_tp_size != decode_tp_size) where replication/head-range computation yields num_heads_to_send whose byte footprint exceeds the destination token capacity, e.g. mismatched total_kv_heads or head_dim between the two sides.

Common situations: Prefill and decode launched with different TP sizes on a model whose kv_heads do not divide evenly, mismatched head_dim/dtype causing different bytes_per_token_dst, or a model whose KV head layout the slice logic does not cover.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/df9c42585c702fce. Report an issue: GitHub.