sgl-project/sglang · error · ValueError

and its host copy must have the same length

Error message

{name} and its host copy must have the same length

What it means

When a host-side copy of cu_seqlens is supplied (cu_seqlens_host) to avoid a device-to-host sync, its length must equal cu_seqlens.numel(). A mismatch means the device tensor and host list describe different numbers of sequences and the kernel would compute wrong boundaries.

Solutions

1. Regenerate the host copy from the same source as the device tensor in the same step: host = cu.tolist() before any batch mutation
2. Ensure both use the inclusive convention: B+1 entries for B sequences
3. Add an assert len(cu_host) == cu.numel() right before the call

Example fix

# before
cu_q = torch.tensor([0,5,12], dtype=torch.int32)
cu_q_host = [0, 5, 12, 20]  # stale, longer
# after
cu_q_host = cu_q.tolist()
assert len(cu_q_host) == cu_q.numel()

if cu_host is not None:
    assert len(cu_host) == cu.numel(), "host copy out of sync with device tensor"

When it happens

Trigger: Calling fused_infer_attention_varlen with cu_seqlens_q_host (or _k_host) whose len() differs from the corresponding device tensor's element count — e.g. host list from a previous batch, or off-by-one (len B vs B+1 boundaries).

Common situations: Caching the host copy across batches and forgetting to update it; building the host copy from per-batch lengths but the device tensor from a different (e.g. filtered) batch; off-by-one boundary conventions.

Related errors

• cu_seqlens_q and cu_seqlens_k must describe the same batch
• must be non-decreasing
• must start with 0 and contain at least one sequence
• n must be a positive power of 2, got
• is required for NPU packed attention

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/da335ec716fa139a. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:33

logger = init_logger(__name__)


def _packed_boundaries(
    cu_seqlens: torch.Tensor,
    cu_seqlens_host: Sequence[int] | None,
    total_tokens: int,
    name: str,
) -> tuple[int, ...]:
    if cu_seqlens is None:
        raise ValueError(f"{name} is required for NPU packed attention")
    if cu_seqlens.ndim != 1 or cu_seqlens.dtype not in (
        torch.int32,
        torch.int64,
    ):
        raise ValueError(f"{name} must be a 1D int32 or int64 tensor")
    if cu_seqlens_host is not None and len(cu_seqlens_host) != cu_seqlens.numel():
        raise ValueError(f"{name} and its host copy must have the same length")

    boundaries = tuple(
        int(value)
        for value in (
            cu_seqlens.tolist() if cu_seqlens_host is None else cu_seqlens_host
        )
    )
    if len(boundaries) < 2 or boundaries[0] != 0:
        raise ValueError(f"{name} must start with 0 and contain at least one sequence")
    if boundaries[-1] != total_tokens:
        raise ValueError(
            f"{name} must end at the packed token count {total_tokens}, "
            f"got {boundaries[-1]}"
        )
    if any(stop < start for start, stop in zip(boundaries[:-1], boundaries[1:])):
        raise ValueError(f"{name} must be non-decreasing")
    return boundaries

View on GitHub (pinned to 0132848349)