sgl-project/sglang · error · ValueError

{name} and its host copy must have the same length

Error message

{name} and its host copy must have the same length

What it means

When a host-side copy of cu_seqlens is supplied (cu_seqlens_host) to avoid a device-to-host sync, its length must equal cu_seqlens.numel(). A mismatch means the device tensor and host list describe different numbers of sequences and the kernel would compute wrong boundaries.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:33

logger = init_logger(__name__)


def _packed_boundaries(
    cu_seqlens: torch.Tensor,
    cu_seqlens_host: Sequence[int] | None,
    total_tokens: int,
    name: str,
) -> tuple[int, ...]:
    if cu_seqlens is None:
        raise ValueError(f"{name} is required for NPU packed attention")
    if cu_seqlens.ndim != 1 or cu_seqlens.dtype not in (
        torch.int32,
        torch.int64,
    ):
        raise ValueError(f"{name} must be a 1D int32 or int64 tensor")
    if cu_seqlens_host is not None and len(cu_seqlens_host) != cu_seqlens.numel():
        raise ValueError(f"{name} and its host copy must have the same length")

    boundaries = tuple(
        int(value)
        for value in (
            cu_seqlens.tolist() if cu_seqlens_host is None else cu_seqlens_host
        )
    )
    if len(boundaries) < 2 or boundaries[0] != 0:
        raise ValueError(f"{name} must start with 0 and contain at least one sequence")
    if boundaries[-1] != total_tokens:
        raise ValueError(
            f"{name} must end at the packed token count {total_tokens}, "
            f"got {boundaries[-1]}"
        )
    if any(stop < start for start, stop in zip(boundaries[:-1], boundaries[1:])):
        raise ValueError(f"{name} must be non-decreasing")
    return boundaries

View on GitHub (pinned to 0132848349)

Solutions

1. Regenerate the host copy from the same source as the device tensor in the same step: host = cu.tolist() before any batch mutation
2. Ensure both use the inclusive convention: B+1 entries for B sequences
3. Add an assert len(cu_host) == cu.numel() right before the call

Example fix

# before
cu_q = torch.tensor([0,5,12], dtype=torch.int32)
cu_q_host = [0, 5, 12, 20]  # stale, longer
# after
cu_q_host = cu_q.tolist()
assert len(cu_q_host) == cu_q.numel()

if cu_host is not None:
    assert len(cu_host) == cu.numel(), "host copy out of sync with device tensor"

When it happens

Trigger: Calling fused_infer_attention_varlen with cu_seqlens_q_host (or _k_host) whose len() differs from the corresponding device tensor's element count — e.g. host list from a previous batch, or off-by-one (len B vs B+1 boundaries).

Common situations: Caching the host copy across batches and forgetting to update it; building the host copy from per-batch lengths but the device tensor from a different (e.g. filtered) batch; off-by-one boundary conventions.

Related errors

• {name} must start with 0 and contain at least one sequence
• {name} must be non-decreasing
• cu_seqlens_q and cu_seqlens_k must describe the same batch
• {name} is required for NPU packed attention
• {name} must be a 1D int32 or int64 tensor

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/da335ec716fa139a. Report an issue: GitHub.