sgl-project/sglang · error · TypeError

num_token_non_padded must be a torch.Tensor

Error message

num_token_non_padded must be a torch.Tensor

What it means

A TypeError raised by the input validation inside _fill_padded_rows. The kernel reads a routing count scalar directly from device memory, so num_token_non_padded must be a 1-element torch.Tensor; passing a Python int or anything else fails this check before launching the Triton kernel.

Source

Thrown at python/sglang/kernels/ops/moe/fill_padded_rows.py:57

    x: torch.Tensor,
    num_token_non_padded: torch.Tensor,
    fill_value,
) -> None:
    """Set ``x[row, :] = fill_value`` for every padded row (row index
    ``>= num_token_non_padded``) using a single Triton launch.

    Replaces the eager ``arange + (>=) + boolean index_put_`` sequence, which
    issues several launch-latency-bound kernels per call. The grid is static
    (one program per row) and the pad count is read from device memory inside
    the kernel, so this is safe to capture inside a CUDA/HIP graph.
    """
    # Metadata-only checks (no device sync): the kernel reads a single scalar
    # routing count from device memory, so it must be a 1-element integer tensor
    # on the same device as ``x``. Use explicit raises (not asserts) so the
    # checks survive ``python -O`` and invalid inputs fail loudly instead of
    # turning into opaque Triton/memory errors.
    if not isinstance(num_token_non_padded, torch.Tensor):
        raise TypeError("num_token_non_padded must be a torch.Tensor")
    if num_token_non_padded.numel() != 1:
        raise ValueError(
            "num_token_non_padded must be a single-element tensor, got shape "
            f"{tuple(num_token_non_padded.shape)}"
        )
    if num_token_non_padded.dtype.is_floating_point:
        raise TypeError(
            "num_token_non_padded must be an integer tensor, got "
            f"{num_token_non_padded.dtype}"
        )
    if num_token_non_padded.device != x.device:
        raise ValueError("num_token_non_padded and x must be on the same device")
    n_rows, n_cols = x.shape
    _fill_padded_rows_kernel[(n_rows,)](
        x,
        num_token_non_padded,
        n_cols,
        fill_value,

View on GitHub (pinned to 0132848349)

Solutions

1. Wrap the count in a tensor: torch.tensor(n, dtype=torch.int32, device=x.device)
2. Check the caller supplying the value — the scheduler usually already carries it as a device tensor; use that instead of a host int
3. If a host int is all you have, accept the (small) sync and move it to device with .to(x.device)

Example fix

// before
_mask_topk_ids_padded_region(topk_ids, n, fill_value=-1)
// after
n_t = torch.tensor(n, dtype=torch.int32, device=topk_ids.device)
_mask_topk_ids_padded_region(topk_ids, n_t, fill_value=-1)

def as_pad_count(v, device):
    if not isinstance(v, torch.Tensor):
        v = torch.tensor(v, dtype=torch.int32, device=device)
    return v

def is_valid_pad_count(v) -> bool:
    return isinstance(v, torch.Tensor) and v.numel() == 1 and not v.dtype.is_floating_point

When it happens

Trigger: Calling _mask_topk_ids_padded_region or _zero_topk_weights_padded_region (or _fill_padded_rows directly) with a Python int/float/numpy scalar for num_token_non_padded instead of a device-resident tensor.

Common situations: Refactoring a MoE padding path where the token count used to be a host-side int; passing cpu_tensor.item() results; porting test code that lazily passes a plain number.

Related errors

• num_token_non_padded must be a single-element tensor, got sh
• `mixed_qkv` must be a 2D tensor (got ndim={mixed_qkv.ndim}).
• num_token_non_padded must be an integer tensor, got {num_tok
• Unsupported activation: {ACTIVATION_TYPE}
• topk kernels only support k <= 32: {k=}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/635dc157b3659b27. Report an issue: GitHub.