{"record":{"id":"91e2b6fb85fe088b","repo":"sgl-project/sglang","slug":"invalid-k-out-device-dtype-for-fused-kv-materializ","errorCode":null,"errorMessage":"Invalid k_out device/dtype for fused KV materialization: got device={k_out.device}, dtype={k_out.dtype}, expected device={kv.device}, dtype={kv.dtype}.","messagePattern":"Invalid k_out device/dtype for fused KV materialization: got device=(.+?), dtype=(.+?), expected device=(.+?), dtype=(.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/speculative/fused_kv_materialize.py","lineNumber":192,"sourceCode":"    half_rotary_dim = rotary_dim // 2\n    BLOCK_HD = triton.next_power_of_2(head_dim)\n\n    if positions.device != kv.device:\n        positions = positions.to(device=kv.device, dtype=torch.int64)\n    elif positions.dtype != torch.int64:\n        positions = positions.to(torch.int64)\n\n    expected_shape = (n_layers, total_ctx, num_kv_heads, head_dim)\n    if k_out is None:\n        k_out = torch.empty(expected_shape, dtype=kv.dtype, device=kv.device)\n    else:\n        if k_out.shape != expected_shape:\n            raise ValueError(\n                \"Invalid k_out shape for fused KV materialization: \"\n                f\"got {tuple(k_out.shape)}, expected {expected_shape}.\"\n            )\n        if k_out.device != kv.device or k_out.dtype != kv.dtype:\n            raise ValueError(\n                \"Invalid k_out device/dtype for fused KV materialization: \"\n                f\"got device={k_out.device}, dtype={k_out.dtype}, \"\n                f\"expected device={kv.device}, dtype={kv.dtype}.\"\n            )\n    if v_out is None:\n        v_out = torch.empty_like(k_out)\n    else:\n        if v_out.shape != expected_shape:\n            raise ValueError(\n                \"Invalid v_out shape for fused KV materialization: \"\n                f\"got {tuple(v_out.shape)}, expected {expected_shape}.\"\n            )\n        if v_out.device != kv.device or v_out.dtype != kv.dtype:\n            raise ValueError(\n                \"Invalid v_out device/dtype for fused KV materialization: \"\n                f\"got device={v_out.device}, dtype={v_out.dtype}, \"\n                f\"expected device={kv.device}, dtype={kv.dtype}.\"\n            )","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/speculative/fused_kv_materialize.py#L174-L210","documentation":"A caller-supplied k_out must live on the same device and have the same dtype as the kv input tensor, since the Triton kernel writes in-place with no conversion.","triggerScenarios":"Passing a CPU buffer while kv is on GPU, or a float16 buffer with bfloat16 kv.","commonSituations":"Preallocating output buffers at module init on the wrong device, or after moving the model with .to(device) without moving the persistent buffer.","solutions":["Allocate k_out with device=kv.device, dtype=kv.dtype or via torch.empty_like semantics.","If None is passed, the function allocates correctly — prefer that.","Move persistent buffers together with the model using register_buffer so .to() tracks them."],"exampleFix":"// before\nk_out = torch.empty(shape, dtype=torch.float16)  # kv is bf16\n// after\nk_out = None  # or torch.empty(shape, dtype=kv.dtype, device=kv.device)","handlingStrategy":"validation","validationCode":"assert k_out is None or (k_out.device == kv.device and k_out.dtype == kv.dtype)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Allocate outputs with device=kv.device, dtype=kv.dtype.","Use register_buffer so .to()/half() moves buffers with the model."],"tags":["device-dtype-validation","kv-cache"],"backgroundTag":"device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}