{"record":{"id":"7a7a205298e4b820","repo":"sgl-project/sglang","slug":"all-inputs-must-be-on-the-same-device-7a7a20","errorCode":null,"errorMessage":"All inputs must be on the same device.","messagePattern":"All inputs must be on the same device\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/helion/kda_decode.py","lineNumber":265,"sourceCode":"            f\"(got ndim={ssm_state_indices.ndim}).\"\n        )\n    if not out.is_contiguous():\n        raise ValueError(\"`out` must be contiguous.\")\n\n    device = mixed_qkv.device\n    if any(\n        tensor.device != device\n        for tensor in (\n            a,\n            b,\n            A_log,\n            dt_bias,\n            initial_state,\n            out,\n            ssm_state_indices,\n        )\n    ):\n        raise ValueError(\"All inputs must be on the same device.\")\n\n    B = mixed_qkv.shape[0]\n    if a.shape[0] != B or b.shape[0] != B:\n        raise ValueError(\n            \"Mismatched batch sizes: \"\n            f\"mixed_qkv.shape[0]={B}, a.shape[0]={a.shape[0]}, \"\n            f\"b.shape[0]={b.shape[0]}.\"\n        )\n    if ssm_state_indices.shape[0] != B:\n        raise ValueError(\n            f\"`ssm_state_indices` must have shape [B] \"\n            f\"(got {tuple(ssm_state_indices.shape)}; expected ({B},)).\"\n        )\n\n    if initial_state.ndim != 4:\n        raise ValueError(\n            f\"`initial_state` must be a 4D tensor (got ndim={initial_state.ndim}).\"\n        )","sourceCodeStart":247,"sourceCodeEnd":283,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/helion/kda_decode.py#L247-L283","documentation":"The packed decode kernel launches on a single device and validates that mixed_qkv, a, b, A_log, dt_bias, initial_state, out, and ssm_state_indices all live on the same CUDA device. Any tensor on CPU or a different GPU index triggers this error.","triggerScenarios":"Passing weights (A_log/dt_bias) still on CPU after an incomplete .to(device), or mixing tensors across cuda:0 and cuda:1.","commonSituations":"Incomplete model.to(cuda) migration; state indices created on default device while inputs are sharded; multi-GPU runs with per-device buffers mismatched.","solutions":["Move all tensors to the same device: t = t.to(mixed_qkv.device)","Audit each tensor's .device right before the call in debugging","For TP runs, ensure the state pool and indices are on the correct rank's device"],"exampleFix":"# before\nout = decode(qkv, a, b, A_log_cpu, dt_bias_cpu, ...)\n# after\nout = decode(qkv, a, b, A_log.to(qkv.device), dt_bias.to(qkv.device), ...)","handlingStrategy":"validation","validationCode":"dev = mixed_qkv.device\ntensors = [a, b, A_log, dt_bias, initial_state, out, ssm_state_indices]\nassert all(t.device == dev for t in tensors), 'device mismatch'","typeGuard":"def all_same_device(ref: torch.Tensor, *ts: torch.Tensor) -> bool:\n    return all(t.device == ref.device for t in ts)","tryCatchPattern":null,"preventionTips":["Move weights to device once at init, not per step","Add a device audit log line when debugging multi-GPU issues"],"tags":["kda","mamba","device-mismatch","helion","multi-gpu"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}