sgl-project/sglang · critical · ValueError

Head slice size evaluates to zero

Error message

Head slice size evaluates to zero

What it means

While building the TP (tensor-parallel) slice configuration for KV transfer, the per-head byte slice computed as bytes_per_token_dst // dst_heads_per_rank evaluates to zero. This means the destination token's byte budget divided among its KV heads yields nothing, typically because total_kv_heads is larger than the per-token byte size or bytes_per_token_dst is zero/miscomputed.

Source

Thrown at python/sglang/srt/disaggregation/mori/conn.py:932

        src_item_len = self.kv_args.kv_item_lens[0]
        dst_item_len = peer_info.dst_kv_item_len

        bytes_per_token_src = src_item_len // page_size
        bytes_per_token_dst = dst_item_len // page_size

        prefill_tp_size = self.attn_tp_size
        decode_tp_size = peer_info.decode_tp_size

        total_kv_heads = getattr(self.kv_args, "total_kv_head_num", 0)
        if total_kv_heads <= 0:
            total_kv_heads = self.kv_args.kv_head_num * prefill_tp_size

        src_heads_per_rank = max(1, total_kv_heads // prefill_tp_size)
        dst_heads_per_rank = max(1, total_kv_heads // decode_tp_size)

        bytes_per_head_slice = bytes_per_token_dst // dst_heads_per_rank
        if bytes_per_head_slice == 0:
            raise ValueError("Head slice size evaluates to zero")

        src_replication = max(1, prefill_tp_size // total_kv_heads)

        local_tp_rank = self.kv_args.engine_rank % prefill_tp_size
        dst_tp_rank = peer_info.decode_tp_rank % decode_tp_size

        if prefill_tp_size > decode_tp_size:
            src_head_start = 0
            num_heads_to_send = src_heads_per_rank
            unique_head_idx = local_tp_rank // src_replication
            dst_head_start = (unique_head_idx * src_heads_per_rank) % dst_heads_per_rank
        else:
            src_head_start = (dst_tp_rank * dst_heads_per_rank) % src_heads_per_rank
            num_heads_to_send = dst_heads_per_rank
            dst_head_start = 0

        src_head_slice_offset = src_head_start * bytes_per_head_slice
        dst_head_slice_offset = dst_head_start * bytes_per_head_slice

View on GitHub (pinned to 0132848349)

Solutions

1. Verify total_kv_heads matches the model's num_kv_heads and that decode_tp_size divides it reasonably
2. Check that the decode-side descriptor bytes_per_token_dst (item_len * page_size arithmetic) is nonzero and matches dtype_size * head_dim * heads
3. Ensure prefill and decode use the same dtype and head_dim for the KV cache
4. Log bytes_per_token_dst and dst_heads_per_rank before transfer to confirm the arithmetic inputs

dst_heads = max(1, total_kv_heads // decode_tp_size)
assert bytes_per_token_dst > 0 and bytes_per_token_dst // dst_heads > 0, (
    "bytes_per_token_dst too small for head slicing"
)

When it happens

Trigger: send_kvcache on a TP-mismatched PD setup where bytes_per_token_dst (from the decode-side memory descriptor item length) is smaller than dst_heads_per_rank (total_kv_heads // decode_tp_size), or total_kv_heads exceeds the token byte size, making the integer division zero.

Common situations: Head-dimension or dtype mismatch between prefill and decode memory pool registration, corrupted/zero item_len in the exchanged memory descriptors, or an exotic TP/heads combination where heads outnumber bytes per token (very small head_dim with tiny dtype).

Related errors

• Slice size exceeds destination token capacity for TP slice t
• PD DCP source/destination KV geometry differs: src={src_toke
• Page size mismatch: prefill server has page_size={info.page_
• Destination MLA KV descriptors do not match prefill pp confi
• PD state transfer does not support TP-mismatched non-MLA SWA

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/41fc9eeb7872f72c. Report an issue: GitHub.