sgl-project/sglang · error · TypeError

num_token_non_padded must be an integer tensor, got {num_tok

Error message

num_token_non_padded must be an integer tensor, got {num_token_non_padded.dtype}

What it means

TypeError raised when num_token_non_padded is a floating-point tensor. The routing count is used as an integer row bound inside the Triton kernel, so a float dtype would silently truncate or misinterpret; the check enforces an integer dtype up front.

Source

Thrown at python/sglang/kernels/ops/moe/fill_padded_rows.py:64

    Replaces the eager ``arange + (>=) + boolean index_put_`` sequence, which
    issues several launch-latency-bound kernels per call. The grid is static
    (one program per row) and the pad count is read from device memory inside
    the kernel, so this is safe to capture inside a CUDA/HIP graph.
    """
    # Metadata-only checks (no device sync): the kernel reads a single scalar
    # routing count from device memory, so it must be a 1-element integer tensor
    # on the same device as ``x``. Use explicit raises (not asserts) so the
    # checks survive ``python -O`` and invalid inputs fail loudly instead of
    # turning into opaque Triton/memory errors.
    if not isinstance(num_token_non_padded, torch.Tensor):
        raise TypeError("num_token_non_padded must be a torch.Tensor")
    if num_token_non_padded.numel() != 1:
        raise ValueError(
            "num_token_non_padded must be a single-element tensor, got shape "
            f"{tuple(num_token_non_padded.shape)}"
        )
    if num_token_non_padded.dtype.is_floating_point:
        raise TypeError(
            "num_token_non_padded must be an integer tensor, got "
            f"{num_token_non_padded.dtype}"
        )
    if num_token_non_padded.device != x.device:
        raise ValueError("num_token_non_padded and x must be on the same device")
    n_rows, n_cols = x.shape
    _fill_padded_rows_kernel[(n_rows,)](
        x,
        num_token_non_padded,
        n_cols,
        fill_value,
        x.stride(0),
        BLOCK_COLS=triton.next_power_of_2(n_cols),
    )

View on GitHub (pinned to 0132848349)

Solutions

1. Create the tensor with an integer dtype: torch.tensor(n, dtype=torch.int32, device=x.device)
2. Cast an existing tensor: num.to(torch.int32) (only after confirming the value is integral)
3. Audit the producer of the scalar — it should never be a float in the first place

Example fix

// before
num = torch.tensor(64.0, device=x.device)
// after
num = torch.tensor(64, dtype=torch.int32, device=x.device)

if num_token_non_padded.dtype.is_floating_point:
    num_token_non_padded = num_token_non_padded.to(torch.int32)

def is_int_scalar(t: torch.Tensor) -> bool:
    return isinstance(t, torch.Tensor) and t.numel() == 1 and not t.dtype.is_floating_point

When it happens

Trigger: Calling _fill_padded_rows (via _mask_topk_ids_padded_region / _zero_topk_weights_padded_region) with num_token_non_padded of dtype float16/bfloat16/float32/float64, e.g. torch.tensor(64.0, device=...).

Common situations: Reusing a tensor produced by a float computation or an averaged count; default dtype set to float via torch.set_default_dtype; converting from a model intermediate without casting.

Related errors

• num_token_non_padded must be a single-element tensor, got sh
• topk_ids must be int32, got {topk_ids.dtype}
• fuse_swiglu_interleaved set on an incompatible fused_moe cal
• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp
• unified_kv dtype mismatch: kv={unified_kv.dtype}, q={q.dtype

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c1b0354faa454b17. Report an issue: GitHub.