sgl-project/sglang · error · ValueError

q must have shape (s_q, h_q, d_qk), got {tuple(q.shape)}

Error message

q must have shape (s_q, h_q, d_qk), got {tuple(q.shape)}

What it means

sparse_mla_q8kv8_prefill_fwd validates argument ranks before unpacking shapes. q must be a 3-D tensor (s_q, h_q, d_qk); passing any other rank raises ValueError with the actual shape so malformed calls fail clearly instead of with an unpacking exception.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:299

    max_logits: Optional[torch.Tensor] = None,  # [s_q, h_q], float32
    lse: Optional[torch.Tensor] = None,  # [s_q, h_q], float32
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """Run Q8KV8 (FP8) sparse prefill attention on SM90.

    The kernel writes into three output tensors. By default fresh tensors
    are allocated and returned; callers that want to reuse buffers may pass
    pre-allocated ``out`` / ``max_logits`` / ``lse`` tensors of the expected
    shape/dtype/device. The three output tensors must not alias each other.

    Returns:
        out:        [s_q, h_q, d_v], bfloat16
        max_logits: [s_q, h_q], float32
        lse:        [s_q, h_q], float32
    """
    # Validate ranks before unpacking shapes so malformed callers fail with a
    # clear error instead of a Python unpacking/indexing exception.
    if q.ndim != 3:
        raise ValueError(f"q must have shape (s_q, h_q, d_qk), got {tuple(q.shape)}")
    if kv.ndim != 3:
        raise ValueError(
            f"kv must have shape (s_kv, h_kv, d_qk), got {tuple(kv.shape)}"
        )
    if indices.ndim != 3:
        raise ValueError(
            "indices must have shape (s_q, h_kv, topk), " f"got {tuple(indices.shape)}"
        )

    s_q, h_q, d_qk = q.shape
    s_kv, h_kv, kv_d_qk = kv.shape
    topk = indices.shape[2]
    device = q.device

    # entry.cuh interprets q/kv as contiguous FP8 buffers and launches all
    # accesses on q's CUDA device. Reject contract violations before launch.
    if not q.is_cuda:
        raise ValueError("q must be a CUDA tensor")

View on GitHub (pinned to 0132848349)

Solutions

1. Reshape q to (s_q, h_q, d_qk) before calling (squeeze/reshape the batch dim into s_q)
2. Check q.ndim == 3 in a wrapper before dispatch

Example fix

# before
q3 = q  # shape [B, S, H, D]
out = sparse_mla_q8kv8_prefill_fwd(q3, kv, indices, ...)
# after
q3 = q.reshape(-1, q.shape[2], q.shape[3])  # [B*S, H, D]
out = sparse_mla_q8kv8_prefill_fwd(q3, kv, indices, ...)

assert q.ndim == 3, f'q must be (s_q, h_q, d_qk), got {q.shape}'

def is_q3d(q: torch.Tensor) -> bool:
    return q.ndim == 3 and q.shape[0] > 0

When it happens

Trigger: Calling sparse_mla_q8kv8_prefill_fwd with a 2-D or 4-D q (e.g. a batched [B,S,H,D] tensor or a flattened [N*D] tensor).

Common situations: Adapters that assume a batched attention layout; feeding the full attention hidden states instead of the reshaped per-head q; mismatches with other APIs that take 4-D qkv.

Related errors

• kv must have shape (s_kv, h_kv, d_qk), got {tuple(kv.shape)}
• {name} must have shape {tuple(shape)}, got {tuple(t.shape)}
• indices must have shape (s_q, h_kv, topk), got {tuple(indice
• Inkling shared-sink LoRA rank dimensions do not match
• The pointers must be multiple of 16 bytes.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5292e044cc11c417. Report an issue: GitHub.