{"record":{"id":"b48b587646a36a72","repo":"sgl-project/sglang","slug":"name-block-cnt-and-name-block-idx-must-be-on-t","errorCode":null,"errorMessage":"{name}_block_cnt and {name}_block_idx must be on the same device","messagePattern":"(.+?)_block_cnt and (.+?)_block_idx must be on the same device","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attn/cute/block_sparsity.py","lineNumber":250,"sourceCode":"def _check_and_expand_block(\n    name: str,\n    cnt: torch.Tensor | None,\n    idx: torch.Tensor | None,\n    expected_count_shape: Tuple[int, ...],\n    expected_index_shape: Tuple[int, ...],\n    context: str | None,\n    hint: str | Callable[[], str] | None,\n) -> Tuple[torch.Tensor | None, torch.Tensor | None]:\n    if (cnt is None) != (idx is None):\n        raise ValueError(\n            f\"{name}_block_cnt and {name}_block_idx must both be provided or both be None\"\n        )\n    if cnt is None or idx is None:\n        return None, None\n    if cnt.dtype != torch.int32 or idx.dtype != torch.int32:\n        raise ValueError(f\"{name}_block tensors must have dtype torch.int32\")\n    if cnt.device != idx.device:\n        raise ValueError(\n            f\"{name}_block_cnt and {name}_block_idx must be on the same device\"\n        )\n    if not cnt.is_cuda or not idx.is_cuda:\n        raise ValueError(f\"{name}_block tensors must live on CUDA\")\n    expanded_cnt = _expand_sparsity_tensor(\n        cnt, expected_count_shape, f\"{name}_block_cnt\", context, hint\n    )\n    # [Note] Allow Compact block sparse indices\n    # Allow the last dimension (n_blocks) of idx to be <= expected, since\n    # FA4 only accesses indices 0..cnt-1 per query tile. This enables compact\n    # index tensors that avoid O(N^2) memory at long sequence lengths.\n    if idx.ndim == 4 and idx.shape[3] <= expected_index_shape[3]:\n        expected_index_shape = (*expected_index_shape[:3], idx.shape[3])\n    expanded_idx = _expand_sparsity_tensor(\n        idx, expected_index_shape, f\"{name}_block_idx\", context, hint\n    )\n    return expanded_cnt, expanded_idx\n","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attn/cute/block_sparsity.py#L232-L268","documentation":"Raised when validating block-sparse attention metadata: the mask/full block count tensor and index tensor for the same group must reside on the same torch device. The library enforces this before expanding/normalizing the tensors for the FA4 cute block-sparse kernels, since the kernel consumes them as paired device pointers.","triggerScenarios":"Calling normalize_block_sparse_tensors (or normalize_block_sparse_config / _bwd) with e.g. mask_block_cnt on cuda:0 and mask_block_idx on cuda:1 or on CPU.","commonSituations":"Tensors created at different times (one from a cached CPU BlockMask conversion, one moved to GPU), or multi-GPU runs where tensors were pinned to different cuda devices.","solutions":["Move both cnt and idx to the same device: idx = idx.to(cnt.device)","Verify no tensor was left on CPU (e.g. built from a numpy array) before passing BlockSparseTensorsTorch","In multi-GPU/TP setups, move all block sparse tensors to the current rank's device"],"exampleFix":"// before\ntensors = BlockSparseTensorsTorch(mask_block_cnt=cnt_cpu, mask_block_idx=idx_cuda)\n// after\ntensors = BlockSparseTensorsTorch(mask_block_cnt=cnt_cpu.to(idx_cuda.device), mask_block_idx=idx_cuda)","handlingStrategy":"validation","validationCode":"dev = mask_block_cnt.device\nassert mask_block_idx.device == dev, f'idx on {mask_block_idx.device}, cnt on {dev}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Move all block sparse tensors to one device in a single prep function","Assert .device equality right after building BlockSparseTensorsTorch"],"tags":["block-sparse","device-mismatch","attention","cuda"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}