{"record":{"id":"98d5b89bf1207a31","repo":"sgl-project/sglang","slug":"invalid-v-out-device-dtype-for-fused-kv-materializ","errorCode":null,"errorMessage":"Invalid v_out device/dtype for fused KV materialization: got device={v_out.device}, dtype={v_out.dtype}, expected device={kv.device}, dtype={kv.dtype}.","messagePattern":"Invalid v_out device/dtype for fused KV materialization: got device=(.+?), dtype=(.+?), expected device=(.+?), dtype=(.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/speculative/fused_kv_materialize.py","lineNumber":206,"sourceCode":"                \"Invalid k_out shape for fused KV materialization: \"\n                f\"got {tuple(k_out.shape)}, expected {expected_shape}.\"\n            )\n        if k_out.device != kv.device or k_out.dtype != kv.dtype:\n            raise ValueError(\n                \"Invalid k_out device/dtype for fused KV materialization: \"\n                f\"got device={k_out.device}, dtype={k_out.dtype}, \"\n                f\"expected device={kv.device}, dtype={kv.dtype}.\"\n            )\n    if v_out is None:\n        v_out = torch.empty_like(k_out)\n    else:\n        if v_out.shape != expected_shape:\n            raise ValueError(\n                \"Invalid v_out shape for fused KV materialization: \"\n                f\"got {tuple(v_out.shape)}, expected {expected_shape}.\"\n            )\n        if v_out.device != kv.device or v_out.dtype != kv.dtype:\n            raise ValueError(\n                \"Invalid v_out device/dtype for fused KV materialization: \"\n                f\"got device={v_out.device}, dtype={v_out.dtype}, \"\n                f\"expected device={kv.device}, dtype={kv.dtype}.\"\n            )\n\n    _fused_norm_rope_kernel_stacked[(total_ctx, num_kv_heads, n_layers)](\n        kv,\n        k_norm_weight,\n        eps,\n        cos_sin_cache,\n        positions,\n        k_out,\n        v_out,\n        kv.stride(0),\n        kv.stride(1),\n        k_norm_weight.stride(0),\n        cos_sin_cache.stride(0),\n        k_out.stride(0),","sourceCodeStart":188,"sourceCodeEnd":224,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/speculative/fused_kv_materialize.py#L188-L224","documentation":"A caller-supplied v_out must be on the same device and have the same dtype as kv; the kernel writes raw values with no casting or transfer.","triggerScenarios":"Passing a CPU or differently-typed v_out buffer (e.g. fp32 while kv is bf16).","commonSituations":"Buffers created before model.to(device)/half(), or mixed precision changes mid-run (e.g. switching to --kv-cache-dtype).","solutions":["Use v_out = None or allocate with kv.dtype/kv.device.","Recreate preallocated buffers whenever the model's dtype or device changes.","Register buffers on the module so device/dtype moves propagate."],"exampleFix":"// before\nv_out = torch.empty(shape)  # fp32 CPU\n// after\nv_out = None","handlingStrategy":"validation","validationCode":"assert v_out is None or (v_out.device == kv.device and v_out.dtype == kv.dtype)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Recreate preallocated buffers whenever dtype/device changes (e.g. kv cache dtype switches)."],"tags":["device-dtype-validation","kv-cache"],"backgroundTag":"device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}