xai-org/x-algorithm · error · ValueError

{name} must live on CUDA

Error message

{name} must live on CUDA

What it means

Raised by _check_and_expand_metadata_tensor when a metadata tensor passed the device check equality but is still not on CUDA — practically hit when the shared device itself is CPU (all tensors are co-located but on CPU). The kernels are CUDA-only.

Source

Thrown at phoenix/xrex/cutedsl/ranker_fa4/block_sparsity.py:271

    return expanded_cnt, expanded_idx


def _check_and_expand_metadata_tensor(
    name: str,
    tensor: torch.Tensor | None,
    expected_shape: Tuple[int, ...],
    context: str | None,
    hint: str | Callable[[], str] | None,
    device: torch.device,
) -> torch.Tensor | None:
    if tensor is None:
        return None
    if tensor.dtype != torch.int32:
        raise ValueError(f"{name} must have dtype torch.int32")
    if tensor.device != device:
        raise ValueError(f"{name} must be on the same device as block sparse tensors")
    if not tensor.is_cuda:
        raise ValueError(f"{name} must live on CUDA")
    return _expand_sparsity_tensor(tensor, expected_shape, name, context, hint)


def get_block_sparse_expected_shapes(
    batch_size: int,
    num_head: int,
    seqlen_q: int,
    seqlen_k: int,
    m_block_size: int,
    n_block_size: int,
    q_stage: int,
) -> Tuple[Tuple[int, int, int], Tuple[int, int, int, int]]:
    m_block_size_effective = q_stage * m_block_size
    expected_m_blocks = ceildiv(seqlen_q, m_block_size_effective)
    expected_n_blocks = ceildiv(seqlen_k, n_block_size)
    expected_count_shape = (batch_size, num_head, expected_m_blocks)
    expected_index_shape = (batch_size, num_head, expected_m_blocks, expected_n_blocks)
    return expected_count_shape, expected_index_shape

View on GitHub (pinned to 24c60942c5)

Solutions

1. Move the whole workload (and all tensors) to a CUDA device
2. If no GPU is available, use a non-block-sparse or CPU-compatible code path instead
3. Check torch.cuda.is_available() before entering the block-sparse path

Example fix

# before
device = torch.device('cpu')
cnt = cnt.to(device); idx = idx.to(device); meta = meta.to(device)

# after
device = torch.device('cuda')
cnt = cnt.to(device); idx = idx.to(device); meta = meta.to(device)

assert torch.cuda.is_available() and meta.is_cuda and mask_block_cnt.is_cuda

def cuda_pipeline_ready(*ts) -> bool:
    return torch.cuda.is_available() and all(t is None or t.is_cuda for t in ts)

When it happens

Trigger: Running the whole pipeline on CPU: block tensors and metadata all on 'cpu', so tensor.device == device passes, but tensor.is_cuda is False.

Common situations: CI or unit tests without GPUs; environments where CUDA_VISIBLE_DEVICES is empty; falling back to CPU execution for a CUDA-only library.

Related errors

• {name}_block tensors must live on CUDA
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name} must have dtype torch.int32
• {name} must be on the same device as block sparse tensors
• Block sparse tensors{context} must have shapes (B, H, M) and

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/ea6374dcb921f6a1. Report an issue: GitHub.