{"record":{"id":"bc4fa5bc677c022d","repo":"sgl-project/sglang","slug":"dst-and-src-must-be-on-the-same-device-dst-devic","errorCode":null,"errorMessage":"dst and src must be on the same device. {dst.device=} {src.device=}","messagePattern":"dst and src must be on the same device\\. (.+?) (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/mamba/mamba_state_scatter_triton.py","lineNumber":248,"sourceCode":"    This function fuses the following operations into a single kernel:\n    1. valid_mask = step_indices_raw >= 0\n    2. valid_indices = valid_mask.nonzero()\n    3. dst_indices = dst_indices_raw[valid_indices]  (index_select)\n    4. step_indices = step_indices_raw[valid_indices]  (index_select)\n    5. for each valid i: dst[:, dst_indices[i], :] = src[:, i, step_indices[i], :]\n\n    Args:\n        dst: Destination tensor [num_layers, cache_size, *state_shape]\n        src: Source tensor [num_layers, spec_size, draft_tokens, *state_shape]\n        dst_indices_raw: Raw destination indices for all requests [total_requests]\n        step_indices_raw: Raw step indices; entry >= 0 means valid [total_requests]\n    \"\"\"\n    total_requests = step_indices_raw.shape[0]\n    if total_requests == 0:\n        return\n\n    if dst.device != src.device:\n        raise ValueError(\n            f\"dst and src must be on the same device. {dst.device=} {src.device=}\"\n        )\n    if not dst.is_cuda or not src.is_cuda:\n        raise ValueError(\n            \"fused_mamba_state_scatter_with_mask only supports CUDA tensors.\"\n        )\n    if dst.ndim < 2 or src.ndim < 3:\n        raise ValueError(f\"Unexpected tensor ranks: {dst.ndim=} {src.ndim=}\")\n    if dst.shape[0] != src.shape[0]:\n        raise ValueError(\n            f\"Layer dimension mismatch: {dst.shape[0]=} vs {src.shape[0]=}\"\n        )\n    if dst.shape[2:] != src.shape[3:]:\n        raise ValueError(\n            f\"Trailing dims mismatch: {dst.shape[2:]=} vs {src.shape[3:]=}\"\n        )\n    if dst_indices_raw.ndim != 1 or step_indices_raw.ndim != 1:\n        raise ValueError(","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/mamba/mamba_state_scatter_triton.py#L230-L266","documentation":"fused_mamba_state_scatter_with_mask requires dst and src on the same CUDA device; a cross-device pair (CPU+GPU, or cuda:0+cuda:1) raises immediately, followed by a CUDA-only check. The kernel issues a device-side copy so devices must match.","triggerScenarios":"Calling fused_mamba_state_scatter_with_mask (or via scatter_mamba_states_after_mtp_verify) with src on cuda:1 and dst on cuda:0, or src still on CPU after capture from another process/GPU.","commonSituations":"Multi-GPU MTP verification where src tensors are gathered from a different rank/device; IPC tensor transfer leaving tensors on the wrong device; CPU-built src buffers in tests.","solutions":["Move src to dst's device: src = src.to(dst.device) before the call","In multi-GPU pipelines, verify device placement of verification outputs before scattering into the cache pool","Assert dst.device == src.device and dst.is_cuda in debug builds"],"exampleFix":"// before\nfused_mamba_state_scatter_with_mask(dst=pool, src=src_on_other_gpu, ...)\n\n// after\nfused_mamba_state_scatter_with_mask(dst=pool, src=src_on_other_gpu.to(pool.device), ...)","handlingStrategy":"type-guard","validationCode":"if src.device != dst.device:\n    src = src.to(dst.device)\nassert dst.is_cuda and src.is_cuda","typeGuard":"def same_cuda_device(a: torch.Tensor, b: torch.Tensor) -> bool:\n    return a.is_cuda and b.is_cuda and a.device == b.device","tryCatchPattern":null,"preventionTips":["In multi-GPU MTP flows, verify rank/device of verification outputs before scatter","Standardize one device per pipeline stage and move tensors at stage boundaries"],"tags":["device-mismatch","mamba","scatter","cuda"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}