sgl-project/sglang · error · ValueError

kv-canary: {name} must have dtype {dtype}, got {tensor.dtype

Error message

kv-canary: {name} must have dtype {dtype}, got {tensor.dtype}

What it means

Plan-kernel input tensors must have exactly the expected dtype (typically int32/int64) because the JIT kernels read raw typed memory without conversion. Any other dtype causes hard-to-debug garbage rather than an async CUDA fault.

Source

Thrown at python/sglang/kernels/ops/kv_canary/plan/utils.py:27

def _resolve_swa_lut(
    lut: Optional[torch.Tensor], device: torch.device
) -> tuple[torch.Tensor, int, bool]:
    """Return the (tensor, length, has_lut) triple to launch the plan kernel with.

    Triton requires a valid tensor pointer at every kernel-arg slot even when ``HAS_SWA_LUT`` is False, so
    when the caller passes ``None`` we substitute a one-element sentinel tensor and set ``lut_len=0``;
    the kernel's constexpr branch guarantees no dereference happens. Dtype matches the production LUT
    (int64) so Triton ``tl.load`` element typing stays consistent.
    """
    if lut is not None:
        return lut, int(lut.shape[0]), True
    return torch.zeros(1, dtype=torch.int64, device=device), 0, False


def _require_dtype(tensor: torch.Tensor, name: str, dtype: torch.dtype) -> None:
    if tensor.dtype != dtype:
        raise ValueError(
            f"kv-canary: {name} must have dtype {dtype}, got {tensor.dtype}"
        )


def _require_1d(tensor: torch.Tensor, name: str) -> None:
    if tensor.ndim != 1:
        raise ValueError(
            f"kv-canary: {name} must be 1-D, got shape {tuple(tensor.shape)}"
        )


def _require_2d(tensor: torch.Tensor, name: str) -> None:
    if tensor.ndim != 2:
        raise ValueError(
            f"kv-canary: {name} must be 2-D, got shape {tuple(tensor.shape)}"
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Cast the offending tensor to the required dtype before the call: t = t.to(torch.int64) (or int32 per the error text)
2. Standardize dtype conventions where the tensors are produced, not at the launch site
3. Check the error message for the tensor name to find which input is wrong

Example fix

// before
prefix_lens = prefix_lens.to(torch.int32)  # kernel wants int64
launch_plan_offsets_kernel(..., prefix_lens=prefix_lens, ...)
// after
prefix_lens = prefix_lens.to(torch.int64)
launch_plan_offsets_kernel(..., prefix_lens=prefix_lens, ...)

expected = {'req_pool_indices': torch.int64, 'prefix_lens': torch.int64, 'extend_seq_lens': torch.int32}  # match kernel spec
for name, t, dt in inputs:
    assert t.dtype == dt, f"{name}: {t.dtype} != {dt}"

def has_dtype(t: torch.Tensor, dt: torch.dtype) -> bool:
    return t.dtype == dt

When it happens

Trigger: Calling the plan launch path with an input tensor whose dtype differs from the required one — e.g. int32 lens where int64 is required, or vice versa.

Common situations: Upstream code refactored lens/indices from int64 to int32 (or the reverse) for memory savings; mixing tensors created by different components with different dtype conventions.

Related errors

• Validate failed: unsupported dtype: {t.dtype}
• kv-canary: {name} must be 1-D, got shape {tuple(tensor.shape
• kv-canary: {name} must be 2-D, got shape {tuple(tensor.shape
• kv-canary: {name} length must be {expected}, got {actual}
• num_token_non_padded must be an integer tensor, got {num_tok

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b315d0a10f54e30e. Report an issue: GitHub.