sgl-project/sglang · error · ValueError

All block sparse tensors must be on the same device

Error message

All block sparse tensors must be on the same device

What it means

Raised when optional full_block_cnt is provided but sits on a different device than mask_block_cnt. All tensors in the block-sparse bundle must be co-located on one CUDA device.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/block_sparsity.py:472

        context,
        hint,
    )
    if mask_cnt is None or mask_idx is None:
        raise ValueError(
            "mask_block_cnt and mask_block_idx must be provided for block sparsity."
        )

    full_cnt, full_idx = _check_and_expand_block(
        "full",
        tensors.full_block_cnt,
        tensors.full_block_idx,
        expected_count_shape,
        expected_index_shape,
        context,
        hint,
    )
    if full_cnt is not None and mask_cnt.device != full_cnt.device:
        raise ValueError("All block sparse tensors must be on the same device")

    dq_write_order = _check_and_expand_metadata_tensor(
        "dq_write_order",
        tensors.dq_write_order,
        tuple(mask_idx.shape),
        context,
        hint,
        mask_cnt.device,
    )
    dq_write_order_full = _check_and_expand_metadata_tensor(
        "dq_write_order_full",
        tensors.dq_write_order_full,
        tuple(full_idx.shape) if full_idx is not None else expected_index_shape,
        context,
        hint,
        mask_cnt.device,
    )
    spt = tensors.spt

View on GitHub (pinned to 0132848349)

Solutions

1. full_cnt = full_cnt.to(mask_cnt.device)
2. Build all four tensors on the same device (torch.cuda.current_device()) from the start

Example fix

// before
tensors = BlockSparseTensorsTorch(cnt, idx, full_block_cnt=full_cpu)
// after
tensors = BlockSparseTensorsTorch(cnt, idx, full_block_cnt=full_cpu.to(cnt.device))

if tensors.full_block_cnt is not None:
    assert tensors.full_block_cnt.device == tensors.mask_block_cnt.device

When it happens

Trigger: mask tensors on cuda:0 and full_block_cnt on cuda:1 or CPU in normalize_block_sparse_tensors.

Common situations: Adding full-block hints later (e.g. for a fused dense/sparse path) from a cache built on another device.

Related errors

• {name}_block_cnt and {name}_block_idx must be on the same de
• {name} must be on the same device as block sparse tensors
• All inputs must be on the same device.
• {tensor_name}{context_clause} with shape {tensor.shape} cann
• {name}_block_cnt and {name}_block_idx must both be provided

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6076eb65e3778abd. Report an issue: GitHub.