sgl-project/sglang · error · ValueError

dst and src must be on the same device. {dst.device=} {src.d

Error message

dst and src must be on the same device. {dst.device=} {src.device=}

What it means

fused_mamba_state_scatter_with_mask requires dst and src on the same CUDA device; a cross-device pair (CPU+GPU, or cuda:0+cuda:1) raises immediately, followed by a CUDA-only check. The kernel issues a device-side copy so devices must match.

Source

Thrown at python/sglang/kernels/ops/mamba/mamba_state_scatter_triton.py:248

    This function fuses the following operations into a single kernel:
    1. valid_mask = step_indices_raw >= 0
    2. valid_indices = valid_mask.nonzero()
    3. dst_indices = dst_indices_raw[valid_indices]  (index_select)
    4. step_indices = step_indices_raw[valid_indices]  (index_select)
    5. for each valid i: dst[:, dst_indices[i], :] = src[:, i, step_indices[i], :]

    Args:
        dst: Destination tensor [num_layers, cache_size, *state_shape]
        src: Source tensor [num_layers, spec_size, draft_tokens, *state_shape]
        dst_indices_raw: Raw destination indices for all requests [total_requests]
        step_indices_raw: Raw step indices; entry >= 0 means valid [total_requests]
    """
    total_requests = step_indices_raw.shape[0]
    if total_requests == 0:
        return

    if dst.device != src.device:
        raise ValueError(
            f"dst and src must be on the same device. {dst.device=} {src.device=}"
        )
    if not dst.is_cuda or not src.is_cuda:
        raise ValueError(
            "fused_mamba_state_scatter_with_mask only supports CUDA tensors."
        )
    if dst.ndim < 2 or src.ndim < 3:
        raise ValueError(f"Unexpected tensor ranks: {dst.ndim=} {src.ndim=}")
    if dst.shape[0] != src.shape[0]:
        raise ValueError(
            f"Layer dimension mismatch: {dst.shape[0]=} vs {src.shape[0]=}"
        )
    if dst.shape[2:] != src.shape[3:]:
        raise ValueError(
            f"Trailing dims mismatch: {dst.shape[2:]=} vs {src.shape[3:]=}"
        )
    if dst_indices_raw.ndim != 1 or step_indices_raw.ndim != 1:
        raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Move src to dst's device: src = src.to(dst.device) before the call
2. In multi-GPU pipelines, verify device placement of verification outputs before scattering into the cache pool
3. Assert dst.device == src.device and dst.is_cuda in debug builds

Example fix

// before
fused_mamba_state_scatter_with_mask(dst=pool, src=src_on_other_gpu, ...)

// after
fused_mamba_state_scatter_with_mask(dst=pool, src=src_on_other_gpu.to(pool.device), ...)

if src.device != dst.device:
    src = src.to(dst.device)
assert dst.is_cuda and src.is_cuda

def same_cuda_device(a: torch.Tensor, b: torch.Tensor) -> bool:
    return a.is_cuda and b.is_cuda and a.device == b.device

When it happens

Trigger: Calling fused_mamba_state_scatter_with_mask (or via scatter_mamba_states_after_mtp_verify) with src on cuda:1 and dst on cuda:0, or src still on CPU after capture from another process/GPU.

Common situations: Multi-GPU MTP verification where src tensors are gathered from a different rank/device; IPC tensor transfer leaving tensors on the wrong device; CPU-built src buffers in tests.

Related errors

• {name}_block_cnt and {name}_block_idx must be on the same de
• All inputs must be on the same device.
• indices must be on q's device {device}, got {indices.device}
• topk_length must be a CUDA tensor
• topk_length must be on q's device {device}, got {topk_length

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bc4fa5bc677c022d. Report an issue: GitHub.