sgl-project/sglang · error · ValueError

All inputs must be on the same device.

Error message

All inputs must be on the same device.

What it means

The packed decode kernel launches on a single device and validates that mixed_qkv, a, b, A_log, dt_bias, initial_state, out, and ssm_state_indices all live on the same CUDA device. Any tensor on CPU or a different GPU index triggers this error.

Source

Thrown at python/sglang/kernels/ops/attention/helion/kda_decode.py:265

            f"(got ndim={ssm_state_indices.ndim})."
        )
    if not out.is_contiguous():
        raise ValueError("`out` must be contiguous.")

    device = mixed_qkv.device
    if any(
        tensor.device != device
        for tensor in (
            a,
            b,
            A_log,
            dt_bias,
            initial_state,
            out,
            ssm_state_indices,
        )
    ):
        raise ValueError("All inputs must be on the same device.")

    B = mixed_qkv.shape[0]
    if a.shape[0] != B or b.shape[0] != B:
        raise ValueError(
            "Mismatched batch sizes: "
            f"mixed_qkv.shape[0]={B}, a.shape[0]={a.shape[0]}, "
            f"b.shape[0]={b.shape[0]}."
        )
    if ssm_state_indices.shape[0] != B:
        raise ValueError(
            f"`ssm_state_indices` must have shape [B] "
            f"(got {tuple(ssm_state_indices.shape)}; expected ({B},))."
        )

    if initial_state.ndim != 4:
        raise ValueError(
            f"`initial_state` must be a 4D tensor (got ndim={initial_state.ndim})."
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Move all tensors to the same device: t = t.to(mixed_qkv.device)
2. Audit each tensor's .device right before the call in debugging
3. For TP runs, ensure the state pool and indices are on the correct rank's device

Example fix

# before
out = decode(qkv, a, b, A_log_cpu, dt_bias_cpu, ...)
# after
out = decode(qkv, a, b, A_log.to(qkv.device), dt_bias.to(qkv.device), ...)

dev = mixed_qkv.device
tensors = [a, b, A_log, dt_bias, initial_state, out, ssm_state_indices]
assert all(t.device == dev for t in tensors), 'device mismatch'

def all_same_device(ref: torch.Tensor, *ts: torch.Tensor) -> bool:
    return all(t.device == ref.device for t in ts)

When it happens

Trigger: Passing weights (A_log/dt_bias) still on CPU after an incomplete .to(device), or mixing tensors across cuda:0 and cuda:1.

Common situations: Incomplete model.to(cuda) migration; state indices created on default device while inputs are sharded; multi-GPU runs with per-device buffers mismatched.

Related errors

• `mixed_qkv` must be a 2D tensor (got ndim={mixed_qkv.ndim}).
• `mixed_qkv` must be contiguous in the last dim.
• `a` and `b` must be 2D tensors (got a.ndim={a.ndim}, b.ndim=
• `a`/`b` must be contiguous in the last dim.
• `A_log`/`dt_bias` must be 1D tensors.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7a7a205298e4b820. Report an issue: GitHub.