{"record":{"id":"d6b52c23822c3283","repo":"sgl-project/sglang","slug":"name-must-have-dtype-dtype-got-t-dtype","errorCode":null,"errorMessage":"{name} must have dtype {dtype}, got {t.dtype}","messagePattern":"(.+?) must have dtype (.+?), got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":116,"sourceCode":"# torch._C._cuda_getCurrentRawStream returns the cudaStream_t pointer expected\n# by the JIT wrapper. torch._C._cuda_getCurrentStream returns a packed stream\n# id and must not be used here.\n_get_current_stream_raw = torch._C._cuda_getCurrentRawStream\n\n\n# Module-level cache for kernel-write-only output tensors. The active s_q rows\n# are overwritten every call; buffers grow monotonically by device/head shape.\ndef _check_out_buffer(\n    t: torch.Tensor,\n    name: str,\n    shape: tuple,\n    dtype: torch.dtype,\n    device: torch.device,\n) -> None:\n    if tuple(t.shape) != tuple(shape):\n        raise ValueError(f\"{name} must have shape {tuple(shape)}, got {tuple(t.shape)}\")\n    if t.dtype != dtype:\n        raise ValueError(f\"{name} must have dtype {dtype}, got {t.dtype}\")\n    if t.device != device:\n        raise ValueError(f\"{name} must be on device {device}, got {t.device}\")\n    if not t.is_contiguous():\n        raise ValueError(f\"{name} must be contiguous\")\n\n\n# Internal custom-op wrappers so the JIT kernel calls participate in\n# torch.library / torch.compile tracing and kernel-API debug logging.\n# The dispatch_full variant carries the optional attn_sink / topk_length\n# tensors as required args; the public API chooses which op to call.\n@register_custom_op(\n    op_name=\"sparse_mla_q8kv8_prefill\",\n    mutates_args=[\"out\", \"max_logits\", \"lse\"],\n)\ndef _sparse_mla_q8kv8_prefill_op(\n    q: torch.Tensor,\n    kv: torch.Tensor,\n    indices: torch.Tensor,","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L98-L134","documentation":"_check_out_buffer requires each caller-supplied output buffer to have exactly the dtype the SM90 kernel writes (documented per buffer, e.g. bfloat16 outputs, float32 max_logits/lse). A dtype mismatch raises ValueError with the buffer name, expected and actual dtype.","triggerScenarios":"Passing out buffers in the wrong dtype: e.g. fp16 or fp32 out instead of bfloat16, or float16 lse instead of float32.","commonSituations":"Porting code from a kernel variant with fp16 outputs; allocating all outputs with the input dtype for convenience; autocast contexts creating unexpected dtypes.","solutions":["Allocate each output with its documented dtype (out: bfloat16; max_logits/lse: float32, per the docstring)","Match the dtype of the buffers the API itself allocates when out= is omitted"],"exampleFix":"# before\nout = torch.empty(s_q, h_q, d_v, dtype=torch.float16, device='cuda')\n# after\nout = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device='cuda')\nlse = torch.empty(s_q, h_q, dtype=torch.float32, device='cuda')","handlingStrategy":"validation","validationCode":"out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=q.device)\nlse = torch.empty(s_q, h_q, dtype=torch.float32, device=q.device)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Copy dtype choices from the API docstring, not from input dtypes","Assert out-buffer dtypes in unit tests"],"tags":["dtype-validation","output-buffer","sparse-mla","fp8"],"backgroundTag":"output-buffer-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}