sgl-project/sglang · error · ValueError

q/k/v dtypes must match

Error message

q/k/v dtypes must match

What it means

The Metal fused RoPE+pool kernel wrapper validates that the q, k, and v tensors share the same dtype before dispatching to the native kernel. The underlying Metal shader is only compiled for a single dtype per invocation, so mixed dtypes (e.g. fp16 q with bf16 k) are rejected upfront.

Source

Thrown at python/sglang/kernels/aot/python/sgl_kernel/metal.py:96

        raise ValueError(
            "q shape must be [num_tokens, num_qo_heads, head_dim], " f"got {q.shape}"
        )
    if k_shape != (q_shape[0], num_kv_heads, head_dim):
        raise ValueError(
            "k shape must be [num_tokens, num_kv_heads, head_dim], " f"got {k.shape}"
        )
    if v_shape != k_shape:
        raise ValueError(f"v shape must match k shape, got {v.shape} vs {k.shape}")
    if positions_shape != (q_shape[0],) or slots_shape != (q_shape[0],):
        raise ValueError("positions/slots must have one entry per token")
    if k_pool_shape[1:] != (num_kv_heads, head_dim):
        raise ValueError(f"k_pool has incompatible shape {k_pool.shape}")
    if v_pool_shape != k_pool_shape:
        raise ValueError(
            f"v_pool shape must match k_pool shape, got {v_pool.shape} vs {k_pool.shape}"
        )
    if q.dtype != k.dtype or q.dtype != v.dtype:
        raise ValueError("q/k/v dtypes must match")
    if k_pool.dtype != q.dtype or v_pool.dtype != q.dtype:
        raise ValueError("pool dtypes must match q/k/v dtype")

    return _metal.rope_pool_fused(
        q,
        k,
        v,
        positions,
        slots,
        k_pool,
        v_pool,
        head_dim,
        num_qo_heads,
        num_kv_heads,
        float(rope_base),
    )

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure q, k, and v are produced from the same precision projection weights (all bf16 or all fp16)
2. Call q,k,v = q.to(dtype),k.to(dtype),v.to(dtype) with a common dtype before the call
3. Check any upstream code that casts kv pools separately from q

Example fix

// before
rope_pool_fused(q.half(), k, v, ...)
// after
dtype = q.dtype
rope_pool_fused(q, k.to(dtype), v.to(dtype), ...)

assert q.dtype == k.dtype == v.dtype, (q.dtype, k.dtype, v.dtype)

def same_qkv_dtype(q,k,v): return q.dtype is k.dtype is v.dtype

except ValueError as e: if 'dtypes must match' in str(e): q,k,v = (t.to(q.dtype) for t in (q,k,v))

When it happens

Trigger: Calling rope_pool_fused (directly or via _rope_custom_aot) with q.dtype != k.dtype or q.dtype != v.dtype, e.g. q float16 while k/v are bfloat16, or q bf16 with k/v float32.

Common situations: Models where the q projection and kv projections are loaded/cast to different precisions, mixing autocast outputs with manually-cast pools, or passing half-precision q with float pools.

Related errors

• pool dtypes must match q/k/v dtype
• rope_pool_fused expects q/k/v to be 3-D
• rope_pool_fused expects positions/slots to be 1-D
• rope_pool_fused expects pool tensors to be 3-D
• q shape must be [num_tokens, num_qo_heads, head_dim], got {q

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/57f70849fff45bbd. Report an issue: GitHub.