sgl-project/sglang · error · ValueError

Invalid v_out device/dtype for fused KV materialization: got

Error message

Invalid v_out device/dtype for fused KV materialization: got device={v_out.device}, dtype={v_out.dtype}, expected device={kv.device}, dtype={kv.dtype}.

What it means

A caller-supplied v_out must be on the same device and have the same dtype as kv; the kernel writes raw values with no casting or transfer.

Source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:206

                "Invalid k_out shape for fused KV materialization: "
                f"got {tuple(k_out.shape)}, expected {expected_shape}."
            )
        if k_out.device != kv.device or k_out.dtype != kv.dtype:
            raise ValueError(
                "Invalid k_out device/dtype for fused KV materialization: "
                f"got device={k_out.device}, dtype={k_out.dtype}, "
                f"expected device={kv.device}, dtype={kv.dtype}."
            )
    if v_out is None:
        v_out = torch.empty_like(k_out)
    else:
        if v_out.shape != expected_shape:
            raise ValueError(
                "Invalid v_out shape for fused KV materialization: "
                f"got {tuple(v_out.shape)}, expected {expected_shape}."
            )
        if v_out.device != kv.device or v_out.dtype != kv.dtype:
            raise ValueError(
                "Invalid v_out device/dtype for fused KV materialization: "
                f"got device={v_out.device}, dtype={v_out.dtype}, "
                f"expected device={kv.device}, dtype={kv.dtype}."
            )

    _fused_norm_rope_kernel_stacked[(total_ctx, num_kv_heads, n_layers)](
        kv,
        k_norm_weight,
        eps,
        cos_sin_cache,
        positions,
        k_out,
        v_out,
        kv.stride(0),
        kv.stride(1),
        k_norm_weight.stride(0),
        cos_sin_cache.stride(0),
        k_out.stride(0),

View on GitHub (pinned to 0132848349)

Solutions

1. Use v_out = None or allocate with kv.dtype/kv.device.
2. Recreate preallocated buffers whenever the model's dtype or device changes.
3. Register buffers on the module so device/dtype moves propagate.

Example fix

// before
v_out = torch.empty(shape)  # fp32 CPU
// after
v_out = None

assert v_out is None or (v_out.device == kv.device and v_out.dtype == kv.dtype)

When it happens

Trigger: Passing a CPU or differently-typed v_out buffer (e.g. fp32 while kv is bf16).

Common situations: Buffers created before model.to(device)/half(), or mixed precision changes mid-run (e.g. switching to --kv-cache-dtype).

Related errors

• Invalid k_out device/dtype for fused KV materialization: got
• v_cache must be provided
• k_cache can only be None when only_qv=True
• rope_pool_fused expects pool tensors to be 3-D
• k_pool has incompatible shape {k_pool.shape}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/98d5b89bf1207a31. Report an issue: GitHub.