{"record":{"id":"bcd75ed59c11e9e9","repo":"sgl-project/sglang","slug":"all-inputs-must-be-on-the-same-device","errorCode":null,"errorMessage":"All inputs must be on the same device.","messagePattern":"All inputs must be on the same device\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/fla/fused_recurrent.py","lineNumber":308,"sourceCode":"    if a.stride(-1) != 1 or b.stride(-1) != 1:\n        raise ValueError(\"`a`/`b` must be contiguous in the last dim.\")\n    if A_log.ndim != 1 or dt_bias.ndim != 1:\n        raise ValueError(\"`A_log`/`dt_bias` must be 1D tensors.\")\n    if A_log.stride(0) != 1 or dt_bias.stride(0) != 1:\n        raise ValueError(\"`A_log`/`dt_bias` must be contiguous.\")\n    if ssm_state_indices.ndim != 1:\n        raise ValueError(\n            f\"`ssm_state_indices` must be 1D for packed decode (got ndim={ssm_state_indices.ndim}).\"\n        )\n    if not out.is_contiguous():\n        raise ValueError(\"`out` must be contiguous.\")\n\n    dev = mixed_qkv.device\n    if any(\n        t.device != dev\n        for t in (a, b, A_log, dt_bias, initial_state, out, ssm_state_indices)\n    ):\n        raise ValueError(\"All inputs must be on the same device.\")\n\n    B = mixed_qkv.shape[0]\n    if a.shape[0] != B or b.shape[0] != B:\n        raise ValueError(\n            \"Mismatched batch sizes: \"\n            f\"mixed_qkv.shape[0]={B}, a.shape[0]={a.shape[0]}, b.shape[0]={b.shape[0]}.\"\n        )\n    if ssm_state_indices.shape[0] != B:\n        raise ValueError(\n            f\"`ssm_state_indices` must have shape [B] (got {tuple(ssm_state_indices.shape)}; expected ({B},)).\"\n        )\n\n    if initial_state.ndim != 4:\n        raise ValueError(\n            f\"`initial_state` must be a 4D tensor (got ndim={initial_state.ndim}).\"\n        )\n    if initial_state.stride(-1) != 1:\n        raise ValueError(\"`initial_state` must be contiguous in the last dim.\")","sourceCodeStart":290,"sourceCodeEnd":326,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/fla/fused_recurrent.py#L290-L326","documentation":"The packed decode kernel launches on mixed_qkv.device and reads/writes a, b, A_log, dt_bias, initial_state, out, and ssm_state_indices with raw pointers, so every tensor must live on the same CUDA device. The wrapper iterates over all inputs and raises on the first mismatch.","triggerScenarios":"Keeping initial_state/out on cuda:0 while the token inputs are on cuda:1 in a TP>=2 setup; parameters (A_log/dt_bias) left on CPU after a partial .to(device); mixed-precision pipelines where some buffers were moved and others weren't.","commonSituations":"Multi-GPU tensor parallel runs with per-rank device ids; loading model weights to meta/cpu and forgetting SSM state caches; writing a single-device benchmark then running under device_map=auto.","solutions":["Move everything to one device: tensors = [t.to(dev, non_blocking=True) for t in tensors] before the call","In TP setups, pass each rank its own slice already resident on that rank's device; verify with assert all(t.device == dev ...)"],"exampleFix":"# before\nout, s = ...(initial_state=cpu_state, ...)  # cpu_state on 'cpu'\n# after\nout, s = ...(initial_state=cpu_state.to(mixed_qkv.device), ...)","handlingStrategy":"validation","validationCode":"dev = mixed_qkv.device\nfor t in (a, b, A_log, dt_bias, initial_state, out, ssm_state_indices):\n    assert t.device == dev, (t.shape, t.device, dev)","typeGuard":"def all_on_device(dev: torch.device, *ts: torch.Tensor) -> bool:\n    return all(t.device == dev for t in ts)","tryCatchPattern":null,"preventionTips":["Move parameters and state caches to the target device once at setup, not per step","In TP code, derive device from the rank's CUDA_VISIBLE configuration explicitly"],"tags":["fla","fused-recurrent","device-mismatch","multi-gpu"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}