{"record":{"id":"e9e25c349f9d0ddf","repo":"vllm-project/vllm","slug":"a-and-b-must-be-on-the-same-device","errorCode":null,"errorMessage":"`a` and `b` must be on the same device.","messagePattern":"`a` and `b` must be on the same device\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/_custom_ops.py","lineNumber":4100,"sourceCode":"\nif hasattr(torch.ops._qutlass_C, \"fusedQuantizeMxAbsMax\"):\n\n    @register_fake(\"_qutlass_C::fusedQuantizeMxAbsMax\")\n    def _fake_fused_quantize_mx_absmax(\n        a: torch.Tensor, b: torch.Tensor, xh_e2m1: torch.Tensor, xh_e8m0: torch.Tensor\n    ):\n        return xh_e2m1, xh_e8m0\n\n\ndef fusedQuantizeMx(\n    a: torch.Tensor, b: torch.Tensor, *, method: Literal[\"quest\", \"abs_max\"] = \"quest\"\n) -> tuple[torch.Tensor, torch.Tensor]:\n    if a.dim() == 0:\n        raise ValueError(\"`a` must have at least 1 dimension.\")\n    if a.size(-1) % 32 != 0:\n        raise ValueError(f\"last dim of `a` must be divisible by 32, got {a.size(-1)}.\")\n    if b.device != a.device:\n        raise ValueError(\"`a` and `b` must be on the same device.\")\n\n    xh_e2m1 = torch.empty(\n        *a.shape[:-1], a.size(-1) // 2, dtype=torch.uint8, device=a.device\n    )\n\n    rows, cols = a.numel() // a.size(-1), a.size(-1) // 32\n    n_row_blocks = cdiv(rows, 128)\n    n_col_blocks = cdiv(cols, 4)\n    padded_rows = n_row_blocks * 128\n    padded_cols = n_col_blocks * 4\n\n    xh_e8m0 = torch.empty(\n        padded_rows, padded_cols, dtype=torch.float8_e8m0fnu, device=a.device\n    )\n\n    if not hasattr(torch.ops, \"_qutlass_C\"):\n        raise RuntimeError(\n            \"The `_qutlass_C` extension is not loaded. \"","sourceCodeStart":4082,"sourceCodeEnd":4118,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/_custom_ops.py#L4082-L4118","documentation":"fusedQuantizeMx() takes two device-resident tensors (a: the values to quantize, b: e.g. a per-row factor/scale input) and passes both straight into the _qutlass_C CUDA kernel. Mixed devices (a on cuda:0, b on cpu or cuda:1) would crash or silently corrupt inside the kernel, so the wrapper enforces same-device up front.","triggerScenarios":"Calling vllm._custom_ops.fusedQuantizeMx(a, b) where b was created with device='cpu' or on another GPU (cuda:1) while a is on cuda:0; or b deserialized/loaded from safetensors to CPU and never moved.","commonSituations":"Forgetting .to(input.device) on a scale/bias tensor loaded from a checkpoint; multi-GPU tensor-parallel code where b was created on the rank-local device but a was already moved.","solutions":["Move b to a.device: ops.fusedQuantizeMx(a, b.to(a.device))","Create b directly on the target device (torch.ones(..., device=a.device)) instead of on CPU"],"exampleFix":"# before\nb = torch.load(\"scale.pt\")                 # cpu tensor\nq, s = ops.fusedQuantizeMx(a, b)\n# after\nb = torch.load(\"scale.pt\").to(a.device)\nq, s = ops.fusedQuantizeMx(a, b)","handlingStrategy":"validation","validationCode":"assert b.device == a.device, f\"device mismatch: a={a.device} b={b.device}\"","typeGuard":"def same_device(a: torch.Tensor, b: torch.Tensor) -> bool:\n    return a.device == b.device","tryCatchPattern":null,"preventionTips":["Always construct scale/bias tensors with device=a.device","Centralize .to(device) right after checkpoint load"],"tags":["quantization","mx-format","device-mismatch","validation","multi-gpu"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}