sgl-project/sglang · error · ValueError

num_token_non_padded and x must be on the same device

Error message

num_token_non_padded and x must be on the same device

What it means

ValueError raised when num_token_non_padded lives on a different device than x. The kernel reads the scalar in-place during launch, so a CPU scalar (or a tensor on another GPU) would cause an illegal memory access rather than a clean failure.

Source

Thrown at python/sglang/kernels/ops/moe/fill_padded_rows.py:69

    # Metadata-only checks (no device sync): the kernel reads a single scalar
    # routing count from device memory, so it must be a 1-element integer tensor
    # on the same device as ``x``. Use explicit raises (not asserts) so the
    # checks survive ``python -O`` and invalid inputs fail loudly instead of
    # turning into opaque Triton/memory errors.
    if not isinstance(num_token_non_padded, torch.Tensor):
        raise TypeError("num_token_non_padded must be a torch.Tensor")
    if num_token_non_padded.numel() != 1:
        raise ValueError(
            "num_token_non_padded must be a single-element tensor, got shape "
            f"{tuple(num_token_non_padded.shape)}"
        )
    if num_token_non_padded.dtype.is_floating_point:
        raise TypeError(
            "num_token_non_padded must be an integer tensor, got "
            f"{num_token_non_padded.dtype}"
        )
    if num_token_non_padded.device != x.device:
        raise ValueError("num_token_non_padded and x must be on the same device")
    n_rows, n_cols = x.shape
    _fill_padded_rows_kernel[(n_rows,)](
        x,
        num_token_non_padded,
        n_cols,
        fill_value,
        x.stride(0),
        BLOCK_COLS=triton.next_power_of_2(n_cols),
    )

View on GitHub (pinned to 0132848349)

Solutions

1. Construct the tensor on x.device: torch.tensor(n, dtype=torch.int32, device=x.device)
2. Or move it: num = num.to(x.device)
3. For multi-GPU, derive the scalar from a tensor already on the target rank's device instead of a global one

Example fix

// before
num = torch.tensor(n, dtype=torch.int32)
// after
num = torch.tensor(n, dtype=torch.int32, device=x.device)

num = num.to(x.device) if num.device != x.device else num

When it happens

Trigger: Calling the padded-region masking helpers with x on cuda:0 but num_token_non_padded on CPU (e.g. torch.tensor(n) without device=), or on cuda:1 in multi-GPU setups.

Common situations: Forgetting device= when constructing the scalar in new MoE code; single-GPU code moved into a TP/EP multi-GPU run; CUDA graph capture where device mismatch only surfaces at capture time.

Related errors

• topk_ids must be a CUDA tensor
• {name}_block_cnt and {name}_block_idx must be on the same de
• Invalid arch format: {arch_str}
• indices must be on q's device {device}, got {indices.device}
• topk_length must be a CUDA tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c3696057ed89df11. Report an issue: GitHub.