{"record":{"id":"c3696057ed89df11","repo":"sgl-project/sglang","slug":"num-token-non-padded-and-x-must-be-on-the-same-dev","errorCode":null,"errorMessage":"num_token_non_padded and x must be on the same device","messagePattern":"num_token_non_padded and x must be on the same device","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/moe/fill_padded_rows.py","lineNumber":69,"sourceCode":"    # Metadata-only checks (no device sync): the kernel reads a single scalar\n    # routing count from device memory, so it must be a 1-element integer tensor\n    # on the same device as ``x``. Use explicit raises (not asserts) so the\n    # checks survive ``python -O`` and invalid inputs fail loudly instead of\n    # turning into opaque Triton/memory errors.\n    if not isinstance(num_token_non_padded, torch.Tensor):\n        raise TypeError(\"num_token_non_padded must be a torch.Tensor\")\n    if num_token_non_padded.numel() != 1:\n        raise ValueError(\n            \"num_token_non_padded must be a single-element tensor, got shape \"\n            f\"{tuple(num_token_non_padded.shape)}\"\n        )\n    if num_token_non_padded.dtype.is_floating_point:\n        raise TypeError(\n            \"num_token_non_padded must be an integer tensor, got \"\n            f\"{num_token_non_padded.dtype}\"\n        )\n    if num_token_non_padded.device != x.device:\n        raise ValueError(\"num_token_non_padded and x must be on the same device\")\n    n_rows, n_cols = x.shape\n    _fill_padded_rows_kernel[(n_rows,)](\n        x,\n        num_token_non_padded,\n        n_cols,\n        fill_value,\n        x.stride(0),\n        BLOCK_COLS=triton.next_power_of_2(n_cols),\n    )\n","sourceCodeStart":51,"sourceCodeEnd":79,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/moe/fill_padded_rows.py#L51-L79","documentation":"ValueError raised when num_token_non_padded lives on a different device than x. The kernel reads the scalar in-place during launch, so a CPU scalar (or a tensor on another GPU) would cause an illegal memory access rather than a clean failure.","triggerScenarios":"Calling the padded-region masking helpers with x on cuda:0 but num_token_non_padded on CPU (e.g. torch.tensor(n) without device=), or on cuda:1 in multi-GPU setups.","commonSituations":"Forgetting device= when constructing the scalar in new MoE code; single-GPU code moved into a TP/EP multi-GPU run; CUDA graph capture where device mismatch only surfaces at capture time.","solutions":["Construct the tensor on x.device: torch.tensor(n, dtype=torch.int32, device=x.device)","Or move it: num = num.to(x.device)","For multi-GPU, derive the scalar from a tensor already on the target rank's device instead of a global one"],"exampleFix":"// before\nnum = torch.tensor(n, dtype=torch.int32)\n// after\nnum = torch.tensor(n, dtype=torch.int32, device=x.device)","handlingStrategy":"validation","validationCode":"num = num.to(x.device) if num.device != x.device else num","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pass device= when constructing scalars destined for kernels","In TP/EP runs, derive scalars from tensors already on the current rank"],"tags":["moe","device-mismatch","cuda","validation"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}