sgl-project/sglang · error · ValueError

num_token_non_padded must be a single-element tensor, got sh

Error message

num_token_non_padded must be a single-element tensor, got shape {tuple(num_token_non_padded.shape)}

What it means

ValueError raised when num_token_non_padded is a torch.Tensor but does not contain exactly one element. The Triton kernel dereferences a single scalar from device memory, so multi-element tensors (shapes like (n,) or ()) with more than one value are rejected.

Source

Thrown at python/sglang/kernels/ops/moe/fill_padded_rows.py:59

    fill_value,
) -> None:
    """Set ``x[row, :] = fill_value`` for every padded row (row index
    ``>= num_token_non_padded``) using a single Triton launch.

    Replaces the eager ``arange + (>=) + boolean index_put_`` sequence, which
    issues several launch-latency-bound kernels per call. The grid is static
    (one program per row) and the pad count is read from device memory inside
    the kernel, so this is safe to capture inside a CUDA/HIP graph.
    """
    # Metadata-only checks (no device sync): the kernel reads a single scalar
    # routing count from device memory, so it must be a 1-element integer tensor
    # on the same device as ``x``. Use explicit raises (not asserts) so the
    # checks survive ``python -O`` and invalid inputs fail loudly instead of
    # turning into opaque Triton/memory errors.
    if not isinstance(num_token_non_padded, torch.Tensor):
        raise TypeError("num_token_non_padded must be a torch.Tensor")
    if num_token_non_padded.numel() != 1:
        raise ValueError(
            "num_token_non_padded must be a single-element tensor, got shape "
            f"{tuple(num_token_non_padded.shape)}"
        )
    if num_token_non_padded.dtype.is_floating_point:
        raise TypeError(
            "num_token_non_padded must be an integer tensor, got "
            f"{num_token_non_padded.dtype}"
        )
    if num_token_non_padded.device != x.device:
        raise ValueError("num_token_non_padded and x must be on the same device")
    n_rows, n_cols = x.shape
    _fill_padded_rows_kernel[(n_rows,)](
        x,
        num_token_non_padded,
        n_cols,
        fill_value,
        x.stride(0),
        BLOCK_COLS=triton.next_power_of_2(n_cols),

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce to a single element: counts[0], counts.squeeze(), or torch.tensor(total, ...)
2. Verify upstream producer — usually num_token_non_padded comes from the scheduler as a 0-dim or (1,) tensor; fix the source that reshaped it

Example fix

// before
num = torch.tensor([n_real, n_pad], device=x.device)
// after
num = torch.tensor(n_real, dtype=torch.int32, device=x.device)

assert num_token_non_padded.numel() == 1, f"expected scalar, got {num_token_non_padded.shape}"

def is_scalar_count(t: torch.Tensor) -> bool:
    return isinstance(t, torch.Tensor) and t.numel() == 1

When it happens

Trigger: Passing a full per-token padding-count vector or an accidentally broadcast tensor (e.g. shape (1,1) works via numel==1? no — (1,1).numel()==1 passes; shapes like (2,) or (n,) fail) to _fill_padded_rows via _mask_topk_ids_padded_region / _zero_topk_weights_padded_region.

Common situations: Plumbing a whole counts tensor through a helper meant for one scalar; slicing mistakes that keep a dimension; test fixtures constructing tensors with torch.zeros(n) instead of torch.tensor(n).

Related errors

• num_token_non_padded must be a torch.Tensor
• num_token_non_padded must be an integer tensor, got {num_tok
• `dt_bias` must have {HV * K} elements (got {dt_bias.numel()}
• `mixed_qkv` must be 2D (got ndim={mixed_qkv.ndim}).
• `mixed_qkv` must be a 2D tensor (got ndim={mixed_qkv.ndim}).

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/79b0b86522140b4d. Report an issue: GitHub.