sgl-project/sglang · error · ValueError

q shape must be [num_tokens, num_qo_heads, head_dim], got {q

Error message

q shape must be [num_tokens, num_qo_heads, head_dim], got {q.shape}

What it means

After rank checks, rope_pool_fused validates the exact shape of q: it must be [num_tokens, num_qo_heads, head_dim] matching the num_qo_heads and head_dim arguments. This error means q's trailing dimensions disagree with the declared head configuration (the leading dim is always accepted, so the mismatch is in heads or head_dim).

Source

Thrown at python/sglang/kernels/aot/python/sgl_kernel/metal.py:78

    Returns:
        `(q_rot, k_rot, k_pool_new, v_pool_new)`.
    """
    if q.ndim != 3 or k.ndim != 3 or v.ndim != 3:
        raise ValueError("rope_pool_fused expects q/k/v to be 3-D")
    if positions.ndim != 1 or slots.ndim != 1:
        raise ValueError("rope_pool_fused expects positions/slots to be 1-D")
    if k_pool.ndim != 3 or v_pool.ndim != 3:
        raise ValueError("rope_pool_fused expects pool tensors to be 3-D")
    q_shape = tuple(q.shape)
    k_shape = tuple(k.shape)
    v_shape = tuple(v.shape)
    positions_shape = tuple(positions.shape)
    slots_shape = tuple(slots.shape)
    k_pool_shape = tuple(k_pool.shape)
    v_pool_shape = tuple(v_pool.shape)

    if q_shape != (q_shape[0], num_qo_heads, head_dim):
        raise ValueError(
            "q shape must be [num_tokens, num_qo_heads, head_dim], " f"got {q.shape}"
        )
    if k_shape != (q_shape[0], num_kv_heads, head_dim):
        raise ValueError(
            "k shape must be [num_tokens, num_kv_heads, head_dim], " f"got {k.shape}"
        )
    if v_shape != k_shape:
        raise ValueError(f"v shape must match k shape, got {v.shape} vs {k.shape}")
    if positions_shape != (q_shape[0],) or slots_shape != (q_shape[0],):
        raise ValueError("positions/slots must have one entry per token")
    if k_pool_shape[1:] != (num_kv_heads, head_dim):
        raise ValueError(f"k_pool has incompatible shape {k_pool.shape}")
    if v_pool_shape != k_pool_shape:
        raise ValueError(
            f"v_pool shape must match k_pool shape, got {v_pool.shape} vs {k_pool.shape}"
        )
    if q.dtype != k.dtype or q.dtype != v.dtype:
        raise ValueError("q/k/v dtypes must match")

View on GitHub (pinned to 0132848349)

Solutions

1. Check q.shape and ensure num_qo_heads == q.shape[1] and head_dim == q.shape[2]
2. If q is packed as [num_tokens, num_qo_heads*head_dim], reshape: q = q.view(num_tokens, num_qo_heads, head_dim)
3. Verify you did not swap the num_qo_heads and num_kv_heads parameters

Example fix

# before
metal.rope_pool_fused(q, k, v, pos, slots, kp, vp,
    num_qo_heads=num_kv_heads, num_kv_heads=num_qo_heads, head_dim=head_dim)

# after
metal.rope_pool_fused(q, k, v, pos, slots, kp, vp,
    num_qo_heads=num_qo_heads, num_kv_heads=num_kv_heads, head_dim=head_dim)

assert q.shape[1:] == (num_qo_heads, head_dim), (q.shape, num_qo_heads, head_dim)

def q_shape_ok(q, num_qo_heads, head_dim):
    return q.ndim == 3 and q.shape[1] == num_qo_heads and q.shape[2] == head_dim

When it happens

Trigger: Calling rope_pool_fused with num_qo_heads/head_dim arguments that don't match q.shape[1] / q.shape[2], e.g. passing GQA head counts for query heads, or a head_dim from a different model config.

Common situations: Copying config values from a different model (head_dim mismatch); mixing num_qo_heads and num_kv_heads argument order; model refactor where q projection output width changed but call site was not updated.

Related errors

• k shape must be [num_tokens, num_kv_heads, head_dim], got {k
• rope_pool_fused expects q/k/v to be 3-D
• rope_pool_fused expects positions/slots to be 1-D
• rope_pool_fused expects pool tensors to be 3-D
• v shape must match k shape, got {v.shape} vs {k.shape}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/17b22cc52078f0b8. Report an issue: GitHub.