{"record":{"id":"d3b1c9b5e734d17b","repo":"sgl-project/sglang","slug":"unsupported-integer-dtype-dtype","errorCode":null,"errorMessage":"Unsupported integer dtype: {dtype}","messagePattern":"Unsupported integer dtype: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/linear/kda_nvidia_prefill/chunk_fwd.py","lineNumber":86,"sourceCode":"def _get_eqlen_dummies(device, idx_dtype=torch.int64):\n    \"\"\"Returns cached (cu_ct, ci_ct) cute wrappers for eqlen (B+1=2, NT+1=2).\"\"\"\n    key = (device.index if device.index is not None else 0, idx_dtype)\n    if key not in _eqlen_dummy_cache:\n        cu_t = torch.empty(2, dtype=idx_dtype, device=device)\n        ci_t = torch.empty(1, 2, dtype=idx_dtype, device=device)\n        cu_etype = cutlass.Int64 if idx_dtype == torch.int64 else cutlass.Int32\n        _eqlen_dummy_cache[key] = (_ct(cu_t, cu_etype), _ct(ci_t, cu_etype))\n    return _eqlen_dummy_cache[key]\n\n\ndef _cute_int_type(dtype):\n    \"\"\"Map PyTorch integer dtype to CUTLASS element type.\"\"\"\n    if dtype == torch.int32:\n        return cutlass.Int32\n    elif dtype == torch.int64:\n        return cutlass.Int64\n    else:\n        raise ValueError(f\"Unsupported integer dtype: {dtype}\")\n\n\n# ========== Fused K1+K2+K3 compilation cache ==========\n_fused_k123_cache = {}\n# id(cu_seqlens) -> bool. Skips per-call GPU->CPU sync on subsequent calls\n# when the same cu_seqlens tensor is reused (typical training/inference loop).\n_varlen_pure_cache = {}\n# id(cu_seqlens) -> int seqlen, populated alongside _varlen_pure_cache for\n# single-seq cu_seqlens.\n_varlen_single_seqlen_cache = {}\n\n# id(tensor) -> cute_wrapper. The wrappers themselves are stateless views\n# over the tensor's storage, so they remain valid as long as the tensor's\n# data pointer / shape / strides don't change. Caller is expected to reuse\n# the same tensor objects across iterations (typical PyTorch pattern).\n_input_wrap_cache = {}\n\n","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/linear/kda_nvidia_prefill/chunk_fwd.py#L68-L104","documentation":"_cute_int_type maps a PyTorch integer dtype to its CUTLASS equivalent for building the fused K1+K2+3 kernel in the KDA NVIDIA prefill path, supporting only torch.int32 (cutlass.Int32) and torch.int64 (cutlass.Int64). Any other integer dtype — int8, int16, uint8, bool, or a non-integer dtype that reached this helper — raises ValueError. It is called from _launch_fused_k123_inv when converting index tensors (e.g. cu_seqlens/indices) into CUTLASS tensor references.","triggerScenarios":"_launch_fused_k123_inv receiving an index tensor in a dtype other than int32/int64 — e.g. cached_indices stored as torch.uint8/int16 to save memory, or a bool mask accidentally passed where integer indices were expected.","commonSituations":"Memory optimizations that downcast index/position tensors to int8/int16; a new caller passing batch indices in a compact dtype; tensors produced on a different framework version defaulting to an unexpected index dtype.","solutions":["Cast the index tensor to int32 (or int64) before calling the fused prefill: idx = idx.to(torch.int32)","Find where the tensor is created and allocate it with dtype=torch.int32 from the start (cheaper than per-call casts)","If you control the input pipeline, validate dtype early and reject/convert non-int32/64 index tensors"],"exampleFix":"# before\nlaunch(mixed_qkv, cache_indices, cu_seqlens=cu_seqlens)  # cache_indices is torch.int16\n# after\nlaunch(mixed_qkv, cache_indices.to(torch.int32), cu_seqlens=cu_seqlens.to(torch.int32))","handlingStrategy":"type-guard","validationCode":"if cache_indices.dtype not in (torch.int32, torch.int64):\\n    cache_indices = cache_indices.to(torch.int32)","typeGuard":"def cutlass_int_tensor(t: torch.Tensor) -> torch.Tensor:\\n    if t.dtype in (torch.int32, torch.int64):\\n        return t\\n    if not t.dtype.is_floating_point:\\n        return t.to(torch.int32)\\n    raise TypeError(f'expected integer tensor, got {t.dtype}')","tryCatchPattern":null,"preventionTips":["Allocate all index/position tensors as torch.int32 at creation","Never downcast index tensors to int8/int16 for memory savings without a cast-back at kernel boundaries"],"tags":["kda","cutlass","dtype","index-tensor","prefill"],"backgroundTag":"unsupported-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}