sgl-project/sglang · error · ValueError

All block sparse tensors must be on the same device

Error message

All block sparse tensors must be on the same device

What it means

Device consistency check for block sparsity: all block-sparse metadata tensors (cnt/idx and related masks) must be on the same CUDA device. Mixed-device metadata would trigger cross-device copies inside the kernel and is rejected up front.

Solutions

1. full_cnt = full_cnt.to(mask_cnt.device)
2. Build all four tensors on the same device (torch.cuda.current_device()) from the start

Example fix

// before
tensors = BlockSparseTensorsTorch(cnt, idx, full_block_cnt=full_cpu)
// after
tensors = BlockSparseTensorsTorch(cnt, idx, full_block_cnt=full_cpu.to(cnt.device))

if tensors.full_block_cnt is not None:
    assert tensors.full_block_cnt.device == tensors.mask_block_cnt.device

When it happens

Trigger: mask tensors on cuda:0 and full_block_cnt on cuda:1 or CPU in normalize_block_sparse_tensors.

Common situations: Adding full-block hints later (e.g. for a fused dense/sparse path) from a cache built on another device.

Related errors

• _block_cnt and _block_idx must be on the same device
• must be on the same device as block sparse tensors
• All inputs must be on the same device.
• All inputs must be on the same device.
• attn_sink must be on q's device

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6076eb65e3778abd. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/block_sparsity.py:472

        context,
        hint,
    )
    if mask_cnt is None or mask_idx is None:
        raise ValueError(
            "mask_block_cnt and mask_block_idx must be provided for block sparsity."
        )

    full_cnt, full_idx = _check_and_expand_block(
        "full",
        tensors.full_block_cnt,
        tensors.full_block_idx,
        expected_count_shape,
        expected_index_shape,
        context,
        hint,
    )
    if full_cnt is not None and mask_cnt.device != full_cnt.device:
        raise ValueError("All block sparse tensors must be on the same device")

    dq_write_order = _check_and_expand_metadata_tensor(
        "dq_write_order",
        tensors.dq_write_order,
        tuple(mask_idx.shape),
        context,
        hint,
        mask_cnt.device,
    )
    dq_write_order_full = _check_and_expand_metadata_tensor(
        "dq_write_order_full",
        tensors.dq_write_order_full,
        tuple(full_idx.shape) if full_idx is not None else expected_index_shape,
        context,
        hint,
        mask_cnt.device,
    )
    spt = tensors.spt

View on GitHub (pinned to 0132848349)