sgl-project/sglang · error · ValueError

{name}_block tensors must have dtype torch.int32

Error message

{name}_block tensors must have dtype torch.int32

What it means

Block-sparsity *_block_cnt/*_block_idx tensors must be torch.int32. Any other dtype (int64 from default arange/range ops, int16, etc.) is rejected with ValueError.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/block_sparsity.py:248

def _check_and_expand_block(
    name: str,
    cnt: torch.Tensor | None,
    idx: torch.Tensor | None,
    expected_count_shape: Tuple[int, ...],
    expected_index_shape: Tuple[int, ...],
    context: str | None,
    hint: str | Callable[[], str] | None,
) -> Tuple[torch.Tensor | None, torch.Tensor | None]:
    if (cnt is None) != (idx is None):
        raise ValueError(
            f"{name}_block_cnt and {name}_block_idx must both be provided or both be None"
        )
    if cnt is None or idx is None:
        return None, None
    if cnt.dtype != torch.int32 or idx.dtype != torch.int32:
        raise ValueError(f"{name}_block tensors must have dtype torch.int32")
    if cnt.device != idx.device:
        raise ValueError(
            f"{name}_block_cnt and {name}_block_idx must be on the same device"
        )
    if not cnt.is_cuda or not idx.is_cuda:
        raise ValueError(f"{name}_block tensors must live on CUDA")
    expanded_cnt = _expand_sparsity_tensor(
        cnt, expected_count_shape, f"{name}_block_cnt", context, hint
    )
    # [Note] Allow Compact block sparse indices
    # Allow the last dimension (n_blocks) of idx to be <= expected, since
    # FA4 only accesses indices 0..cnt-1 per query tile. This enables compact
    # index tensors that avoid O(N^2) memory at long sequence lengths.
    if idx.ndim == 4 and idx.shape[3] <= expected_index_shape[3]:
        expected_index_shape = (*expected_index_shape[:3], idx.shape[3])
    expanded_idx = _expand_sparsity_tensor(
        idx, expected_index_shape, f"{name}_block_idx", context, hint
    )

View on GitHub (pinned to 0132848349)

Solutions

1. Cast both tensors: cnt.to(torch.int32), idx.to(torch.int32)
2. Create with explicit dtype: torch.arange(..., dtype=torch.int32, device='cuda')

Example fix

# before
idx = torch.arange(nblocks)  # int64
# after
idx = torch.arange(nblocks, dtype=torch.int32, device='cuda')

if cnt is not None:\n    cnt = cnt.to(torch.int32); idx = idx.to(torch.int32)\nassert cnt is None or (cnt.is_cuda and idx.is_cuda and cnt.device == idx.device)

def valid_block_dtype(t: torch.Tensor) -> bool:\n    return t.dtype == torch.int32

When it happens

Trigger: Passing metadata tensors created as torch.long (default for arange/tensor(list)) or other dtypes to normalize_block_sparse_tensors.

Common situations: Metadata built with torch.arange(...) (defaults to int64) or from numpy int64 arrays converted to torch; porting from kernels that accept int64.

Related errors

• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp
• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype
• sparse_attn_v4_paged_prefill expects fp16/bf16 q, got {q.dty
• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype
• kv dtype mismatch: kv={kv.dtype}, q={q.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/688e48ae49610eaa. Report an issue: GitHub.