{"record":{"id":"57f70849fff45bbd","repo":"sgl-project/sglang","slug":"q-k-v-dtypes-must-match","errorCode":null,"errorMessage":"q/k/v dtypes must match","messagePattern":"q/k/v dtypes must match","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/aot/python/sgl_kernel/metal.py","lineNumber":96,"sourceCode":"        raise ValueError(\n            \"q shape must be [num_tokens, num_qo_heads, head_dim], \" f\"got {q.shape}\"\n        )\n    if k_shape != (q_shape[0], num_kv_heads, head_dim):\n        raise ValueError(\n            \"k shape must be [num_tokens, num_kv_heads, head_dim], \" f\"got {k.shape}\"\n        )\n    if v_shape != k_shape:\n        raise ValueError(f\"v shape must match k shape, got {v.shape} vs {k.shape}\")\n    if positions_shape != (q_shape[0],) or slots_shape != (q_shape[0],):\n        raise ValueError(\"positions/slots must have one entry per token\")\n    if k_pool_shape[1:] != (num_kv_heads, head_dim):\n        raise ValueError(f\"k_pool has incompatible shape {k_pool.shape}\")\n    if v_pool_shape != k_pool_shape:\n        raise ValueError(\n            f\"v_pool shape must match k_pool shape, got {v_pool.shape} vs {k_pool.shape}\"\n        )\n    if q.dtype != k.dtype or q.dtype != v.dtype:\n        raise ValueError(\"q/k/v dtypes must match\")\n    if k_pool.dtype != q.dtype or v_pool.dtype != q.dtype:\n        raise ValueError(\"pool dtypes must match q/k/v dtype\")\n\n    return _metal.rope_pool_fused(\n        q,\n        k,\n        v,\n        positions,\n        slots,\n        k_pool,\n        v_pool,\n        head_dim,\n        num_qo_heads,\n        num_kv_heads,\n        float(rope_base),\n    )\n","sourceCodeStart":78,"sourceCodeEnd":113,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/aot/python/sgl_kernel/metal.py#L78-L113","documentation":"The Metal fused RoPE+pool kernel wrapper validates that the q, k, and v tensors share the same dtype before dispatching to the native kernel. The underlying Metal shader is only compiled for a single dtype per invocation, so mixed dtypes (e.g. fp16 q with bf16 k) are rejected upfront.","triggerScenarios":"Calling rope_pool_fused (directly or via _rope_custom_aot) with q.dtype != k.dtype or q.dtype != v.dtype, e.g. q float16 while k/v are bfloat16, or q bf16 with k/v float32.","commonSituations":"Models where the q projection and kv projections are loaded/cast to different precisions, mixing autocast outputs with manually-cast pools, or passing half-precision q with float pools.","solutions":["Ensure q, k, and v are produced from the same precision projection weights (all bf16 or all fp16)","Call q,k,v = q.to(dtype),k.to(dtype),v.to(dtype) with a common dtype before the call","Check any upstream code that casts kv pools separately from q"],"exampleFix":"// before\nrope_pool_fused(q.half(), k, v, ...)\n// after\ndtype = q.dtype\nrope_pool_fused(q, k.to(dtype), v.to(dtype), ...)","handlingStrategy":"validation","validationCode":"assert q.dtype == k.dtype == v.dtype, (q.dtype, k.dtype, v.dtype)","typeGuard":"def same_qkv_dtype(q,k,v): return q.dtype is k.dtype is v.dtype","tryCatchPattern":"except ValueError as e: if 'dtypes must match' in str(e): q,k,v = (t.to(q.dtype) for t in (q,k,v))","preventionTips":["Derive q/k/v from the same module dtype","Assert dtypes in test fixtures"],"tags":["metal","rope","dtype-mismatch","apple-silicon"],"backgroundTag":"dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}