{"record":{"id":"3467bec451853625","repo":"sgl-project/sglang","slug":"pool-dtypes-must-match-q-k-v-dtype","errorCode":null,"errorMessage":"pool dtypes must match q/k/v dtype","messagePattern":"pool dtypes must match q/k/v dtype","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/aot/python/sgl_kernel/metal.py","lineNumber":98,"sourceCode":"        )\n    if k_shape != (q_shape[0], num_kv_heads, head_dim):\n        raise ValueError(\n            \"k shape must be [num_tokens, num_kv_heads, head_dim], \" f\"got {k.shape}\"\n        )\n    if v_shape != k_shape:\n        raise ValueError(f\"v shape must match k shape, got {v.shape} vs {k.shape}\")\n    if positions_shape != (q_shape[0],) or slots_shape != (q_shape[0],):\n        raise ValueError(\"positions/slots must have one entry per token\")\n    if k_pool_shape[1:] != (num_kv_heads, head_dim):\n        raise ValueError(f\"k_pool has incompatible shape {k_pool.shape}\")\n    if v_pool_shape != k_pool_shape:\n        raise ValueError(\n            f\"v_pool shape must match k_pool shape, got {v_pool.shape} vs {k_pool.shape}\"\n        )\n    if q.dtype != k.dtype or q.dtype != v.dtype:\n        raise ValueError(\"q/k/v dtypes must match\")\n    if k_pool.dtype != q.dtype or v_pool.dtype != q.dtype:\n        raise ValueError(\"pool dtypes must match q/k/v dtype\")\n\n    return _metal.rope_pool_fused(\n        q,\n        k,\n        v,\n        positions,\n        slots,\n        k_pool,\n        v_pool,\n        head_dim,\n        num_qo_heads,\n        num_kv_heads,\n        float(rope_base),\n    )\n","sourceCodeStart":80,"sourceCodeEnd":113,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/aot/python/sgl_kernel/metal.py#L80-L113","documentation":"The Metal rope_pool_fused wrapper requires the k_pool and v_pool tensors to have the exact same dtype as q/k/v. The fused kernel writes RoPE-rotated and pooled values with a single compiled element type, so mismatched pool dtypes are rejected.","triggerScenarios":"Calling rope_pool_fused where k_pool.dtype or v_pool.dtype differs from q.dtype, e.g. bf16 q with fp32 k_pool, or fp16 v_pool with bf16 q.","commonSituations":"Pre-allocated KV pools in float32 for accumulation while the model runs in bf16/fp16; reusing pools allocated by a different backend; partial casts applied to pools but not q/k/v.","solutions":["Allocate k_pool/v_pool with torch.empty(..., dtype=q.dtype)","Cast pools to q.dtype before the call: k_pool=k_pool.to(q.dtype)","If float32 pooling is required, use a kernel variant that supports it instead of this fused op"],"exampleFix":"// before\nk_pool = torch.zeros(n, dtype=torch.float32)\nrope_pool_fused(q, k, v, k_pool, v_pool, ...)\n// after\nk_pool = torch.zeros(n, dtype=q.dtype)\nrope_pool_fused(q, k, v, k_pool, v_pool, ...)","handlingStrategy":"validation","validationCode":"assert k_pool.dtype == v_pool.dtype == q.dtype","typeGuard":null,"tryCatchPattern":"except ValueError: k_pool = k_pool.to(q.dtype); v_pool = v_pool.to(q.dtype); retry","preventionTips":["Allocate pools with dtype=q.dtype","Centralize pool allocation next to model dtype config"],"tags":["metal","rope","pool","dtype-mismatch"],"backgroundTag":"dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}