sgl-project/sglang · error · ValueError

Unsupported integer dtype: {dtype}

Error message

Unsupported integer dtype: {dtype}

What it means

_cute_int_type maps a PyTorch integer dtype to its CUTLASS equivalent for building the fused K1+K2+3 kernel in the KDA NVIDIA prefill path, supporting only torch.int32 (cutlass.Int32) and torch.int64 (cutlass.Int64). Any other integer dtype — int8, int16, uint8, bool, or a non-integer dtype that reached this helper — raises ValueError. It is called from _launch_fused_k123_inv when converting index tensors (e.g. cu_seqlens/indices) into CUTLASS tensor references.

Source

Thrown at python/sglang/kernels/ops/attention/linear/kda_nvidia_prefill/chunk_fwd.py:86

def _get_eqlen_dummies(device, idx_dtype=torch.int64):
    """Returns cached (cu_ct, ci_ct) cute wrappers for eqlen (B+1=2, NT+1=2)."""
    key = (device.index if device.index is not None else 0, idx_dtype)
    if key not in _eqlen_dummy_cache:
        cu_t = torch.empty(2, dtype=idx_dtype, device=device)
        ci_t = torch.empty(1, 2, dtype=idx_dtype, device=device)
        cu_etype = cutlass.Int64 if idx_dtype == torch.int64 else cutlass.Int32
        _eqlen_dummy_cache[key] = (_ct(cu_t, cu_etype), _ct(ci_t, cu_etype))
    return _eqlen_dummy_cache[key]


def _cute_int_type(dtype):
    """Map PyTorch integer dtype to CUTLASS element type."""
    if dtype == torch.int32:
        return cutlass.Int32
    elif dtype == torch.int64:
        return cutlass.Int64
    else:
        raise ValueError(f"Unsupported integer dtype: {dtype}")


# ========== Fused K1+K2+K3 compilation cache ==========
_fused_k123_cache = {}
# id(cu_seqlens) -> bool. Skips per-call GPU->CPU sync on subsequent calls
# when the same cu_seqlens tensor is reused (typical training/inference loop).
_varlen_pure_cache = {}
# id(cu_seqlens) -> int seqlen, populated alongside _varlen_pure_cache for
# single-seq cu_seqlens.
_varlen_single_seqlen_cache = {}

# id(tensor) -> cute_wrapper. The wrappers themselves are stateless views
# over the tensor's storage, so they remain valid as long as the tensor's
# data pointer / shape / strides don't change. Caller is expected to reuse
# the same tensor objects across iterations (typical PyTorch pattern).
_input_wrap_cache = {}

View on GitHub (pinned to 0132848349)

Solutions

1. Cast the index tensor to int32 (or int64) before calling the fused prefill: idx = idx.to(torch.int32)
2. Find where the tensor is created and allocate it with dtype=torch.int32 from the start (cheaper than per-call casts)
3. If you control the input pipeline, validate dtype early and reject/convert non-int32/64 index tensors

Example fix

# before
launch(mixed_qkv, cache_indices, cu_seqlens=cu_seqlens)  # cache_indices is torch.int16
# after
launch(mixed_qkv, cache_indices.to(torch.int32), cu_seqlens=cu_seqlens.to(torch.int32))

if cache_indices.dtype not in (torch.int32, torch.int64):\n    cache_indices = cache_indices.to(torch.int32)

def cutlass_int_tensor(t: torch.Tensor) -> torch.Tensor:\n    if t.dtype in (torch.int32, torch.int64):\n        return t\n    if not t.dtype.is_floating_point:\n        return t.to(torch.int32)\n    raise TypeError(f'expected integer tensor, got {t.dtype}')

When it happens

Trigger: _launch_fused_k123_inv receiving an index tensor in a dtype other than int32/int64 — e.g. cached_indices stored as torch.uint8/int16 to save memory, or a bool mask accidentally passed where integer indices were expected.

Common situations: Memory optimizations that downcast index/position tensors to int8/int16; a new caller passing batch indices in a compact dtype; tensors produced on a different framework version defaulting to an unexpected index dtype.

Related errors

• sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dty
• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype
• SplitKV partial output (mO) must be Float32
• All tensors must have the same data type
• Only Float16 or BFloat16 is supported

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d3b1c9b5e734d17b. Report an issue: GitHub.