sgl-project/sglang · error · ValueError

{name} must end at the packed token count {total_tokens}, go

Error message

{name} must end at the packed token count {total_tokens}, got {boundaries[-1]}

What it means

_packed_boundaries verifies that the last cu_seqlens entry equals the total token count of the corresponding tensor (q.shape[0] or k.shape[0] in TND layout). A mismatch means the boundaries do not cover every token in the packed tensor — tokens would be silently dropped or over-counted.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/ascend_fa.py:44

        raise ValueError(f"{name} is required for NPU packed attention")
    if cu_seqlens.ndim != 1 or cu_seqlens.dtype not in (
        torch.int32,
        torch.int64,
    ):
        raise ValueError(f"{name} must be a 1D int32 or int64 tensor")
    if cu_seqlens_host is not None and len(cu_seqlens_host) != cu_seqlens.numel():
        raise ValueError(f"{name} and its host copy must have the same length")

    boundaries = tuple(
        int(value)
        for value in (
            cu_seqlens.tolist() if cu_seqlens_host is None else cu_seqlens_host
        )
    )
    if len(boundaries) < 2 or boundaries[0] != 0:
        raise ValueError(f"{name} must start with 0 and contain at least one sequence")
    if boundaries[-1] != total_tokens:
        raise ValueError(
            f"{name} must end at the packed token count {total_tokens}, "
            f"got {boundaries[-1]}"
        )
    if any(stop < start for start, stop in zip(boundaries[:-1], boundaries[1:])):
        raise ValueError(f"{name} must be non-decreasing")
    return boundaries


def fused_infer_attention_varlen(
    q: torch.Tensor,
    k: torch.Tensor,
    v: torch.Tensor,
    cu_seqlens_q: torch.Tensor,
    cu_seqlens_k: torch.Tensor,
    *,
    cu_seqlens_q_host: Sequence[int] | None = None,
    cu_seqlens_k_host: Sequence[int] | None = None,
    softmax_scale: float | None = None,

View on GitHub (pinned to 0132848349)

Solutions

1. Recompute cu_seqlens from the actual tensor: ensure boundaries[-1] == tensor.shape[0]
2. When slicing tokens off the end, also clip the boundary values: cu = torch.clamp(cu, max=new_T) keeping monotonicity
3. For ring-KV chunks, pass chunk-local cu_seqlens_k whose final entry is the chunk's token count

Example fix

# before
q = q[:100]  # truncated
cu_q = torch.tensor([0, 60, 130], dtype=torch.int32)  # ends at 130
# after
q = q[:100]
cu_q = torch.tensor([0, 60, 100], dtype=torch.int32)  # ends at q.shape[0]

assert cu_q[-1].item() == q.shape[0] and cu_k[-1].item() == k.shape[0]

When it happens

Trigger: Passing q with T=100 but cu_seqlens_k ending at 90 (or cu_seqlens_q ending at 120); typically after truncating/masking a packed batch on one side only, or reusing cu_seqlens from a different batch.

Common situations: Padded packed batches where padding tokens were sliced off the tensor but not from the boundaries; batched ring-attention chunking that splits KV without adjusting the KV boundaries; stale metadata reused across iterations.

Related errors

• {name} is required for NPU packed attention
• {name} must start with 0 and contain at least one sequence
• {name} must be non-decreasing
• NPU packed attention requires q, k, and v in [T, N, D] layou
• NPU packed attention requires matching K/V token and head co

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/afe5fa4ca7a575e2. Report an issue: GitHub.