sgl-project/sglang · error · RuntimeError

D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE

Error message

D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}

What it means

The FP8 group quantization used by sparse_attn_v4_paged_decode quantizes the KV head dimension in fixed-size groups (_FP8_GROUP_SIZE). The last dimension of unified_kv must be divisible by that group size, otherwise per-group scales cannot be laid out.

Source

Thrown at python/sglang/kernels/ops/attention/dsv4/unified_kv_kernels/paged_decode.py:671

            "Triton sparse_attn_v4_paged_decode requires CUDA/HIP tensors"
        )
    if q.dtype not in (torch.bfloat16, torch.float16):
        raise RuntimeError(
            f"sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtype}"
        )

    quant_kv = kv_scales is not None
    if quant_kv:
        if unified_kv.dtype != _FP8_DTYPE:
            raise RuntimeError(
                f"kv_scales supplied but unified_kv is {unified_kv.dtype}, "
                f"expected {_FP8_DTYPE}"
            )
        if kv_scales.dtype != torch.float32:
            raise RuntimeError(f"kv_scales must be fp32, got {kv_scales.dtype}")
        D_check = unified_kv.shape[-1]
        if D_check % _FP8_GROUP_SIZE != 0:
            raise RuntimeError(
                f"D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE}"
            )
        expected_g = D_check // _FP8_GROUP_SIZE
        if kv_scales.shape != (unified_kv.shape[0], expected_g):
            raise RuntimeError(
                f"kv_scales shape {tuple(kv_scales.shape)} does not match "
                f"expected ({unified_kv.shape[0]}, {expected_g})"
            )
        if kv_scales.stride(-1) != 1:
            kv_scales = kv_scales.contiguous()
    else:
        if unified_kv.dtype != q.dtype:
            raise RuntimeError(
                f"unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype}"
            )

    T, H, D = q.shape
    out = torch.empty_like(q)

View on GitHub (pinned to 0132848349)

Solutions

1. Check the model's head_dim; if it is not divisible by the FP8 group size, disable FP8 KV quantization for this model
2. Verify the unified_kv cache was built with the intended head_dim (no padding/truncation bug)
3. Pad or reshape the cache last dim to a multiple of _FP8_GROUP_SIZE if your integration allows

FP8_GROUP_SIZE = 128  # keep in sync with kernel
if kv_scales is not None:
    assert unified_kv.shape[-1] % FP8_GROUP_SIZE == 0

When it happens

Trigger: Calling sparse_attn_v4_paged_decode with a unified_kv whose head_dim D is not divisible by _FP8_GROUP_SIZE while kv_scales is provided.

Common situations: A DeepSeek variant with an unusual head_dim (e.g. 192 vs 128) run with FP8 KV quantization enabled; head_dim config mismatch between model and cache.

Related errors

• kv_scales shape {tuple(kv_scales.shape)} does not match expe
• head_dim mismatch: unified_kv={unified_kv.size(-1)}, kv={kv.
• rope_pool_fused expects pool tensors to be 3-D
• k_pool has incompatible shape {k_pool.shape}
• v_pool shape must match k_pool shape, got {v_pool.shape} vs

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f9927a5331c85589. Report an issue: GitHub.