sgl-project/sglang · error · ValueError

{name} must have dtype {dtype}, got {t.dtype}

Error message

{name} must have dtype {dtype}, got {t.dtype}

What it means

_check_out_buffer requires each caller-supplied output buffer to have exactly the dtype the SM90 kernel writes (documented per buffer, e.g. bfloat16 outputs, float32 max_logits/lse). A dtype mismatch raises ValueError with the buffer name, expected and actual dtype.

Source

Thrown at python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py:116

# torch._C._cuda_getCurrentRawStream returns the cudaStream_t pointer expected
# by the JIT wrapper. torch._C._cuda_getCurrentStream returns a packed stream
# id and must not be used here.
_get_current_stream_raw = torch._C._cuda_getCurrentRawStream


# Module-level cache for kernel-write-only output tensors. The active s_q rows
# are overwritten every call; buffers grow monotonically by device/head shape.
def _check_out_buffer(
    t: torch.Tensor,
    name: str,
    shape: tuple,
    dtype: torch.dtype,
    device: torch.device,
) -> None:
    if tuple(t.shape) != tuple(shape):
        raise ValueError(f"{name} must have shape {tuple(shape)}, got {tuple(t.shape)}")
    if t.dtype != dtype:
        raise ValueError(f"{name} must have dtype {dtype}, got {t.dtype}")
    if t.device != device:
        raise ValueError(f"{name} must be on device {device}, got {t.device}")
    if not t.is_contiguous():
        raise ValueError(f"{name} must be contiguous")


# Internal custom-op wrappers so the JIT kernel calls participate in
# torch.library / torch.compile tracing and kernel-API debug logging.
# The dispatch_full variant carries the optional attn_sink / topk_length
# tensors as required args; the public API chooses which op to call.
@register_custom_op(
    op_name="sparse_mla_q8kv8_prefill",
    mutates_args=["out", "max_logits", "lse"],
)
def _sparse_mla_q8kv8_prefill_op(
    q: torch.Tensor,
    kv: torch.Tensor,
    indices: torch.Tensor,

View on GitHub (pinned to 0132848349)

Solutions

  1. Allocate each output with its documented dtype (out: bfloat16; max_logits/lse: float32, per the docstring)
  2. Match the dtype of the buffers the API itself allocates when out= is omitted

Example fix

# before
out = torch.empty(s_q, h_q, d_v, dtype=torch.float16, device='cuda')
# after
out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device='cuda')
lse = torch.empty(s_q, h_q, dtype=torch.float32, device='cuda')
Defensive patterns

Strategy: validation

Validate before calling

out = torch.empty(s_q, h_q, d_v, dtype=torch.bfloat16, device=q.device)
lse = torch.empty(s_q, h_q, dtype=torch.float32, device=q.device)

Prevention

When it happens

Trigger: Passing out buffers in the wrong dtype: e.g. fp16 or fp32 out instead of bfloat16, or float16 lse instead of float32.

Common situations: Porting code from a kernel variant with fp16 outputs; allocating all outputs with the input dtype for convenience; autocast contexts creating unexpected dtypes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d6b52c23822c3283. Report an issue: GitHub.