sgl-project/sglang · error · ValueError

forward_batch with seq_lens is required for TopK retrieval

Error message

forward_batch with seq_lens is required for TopK retrieval

What it means

BaseSparseAlgorithm.retrieve_topk requires the forward_batch keyword argument carrying seq_lens, because Top-K retrieval must gather per-request KV locations from req_to_token using each sequence's length. Without it there is no way to bound the searchable token range per request.

Source

Thrown at python/sglang/srt/mem_cache/sparsity/algorithms/base_algorithm.py:282

        queries: torch.Tensor,
        layer_id: int,
        req_pool_indices: torch.Tensor,
        sparse_mask: torch.Tensor,
        **kwargs,
    ) -> tuple:
        """
        Default TopK retrieval: score-based selection + recent pages.
        Subclasses can override for query-dependent retrieval.

        TODO:
            1. Using triton kernel to speed up this function
            2. Support CUDA Graph
        """
        bs, device = queries.shape[0], queries.device

        seq_lens_source = kwargs.get("forward_batch", None)
        if seq_lens_source is None or not hasattr(seq_lens_source, "seq_lens"):
            raise ValueError(
                "forward_batch with seq_lens is required for TopK retrieval"
            )
        seq_lens = seq_lens_source.seq_lens.to(device)

        req_to_token = self.req_to_token_pool.req_to_token
        max_req_tokens = req_to_token.shape[1]

        per_request_indices = []
        per_request_lengths = []

        for i in range(bs):
            if not sparse_mask[i]:
                per_request_indices.append(
                    torch.empty(0, device=device, dtype=torch.int32)
                )
                per_request_lengths.append(0)
                continue

View on GitHub (pinned to 0132848349)

Solutions

1. Pass forward_batch through: retrieve_topk(queries, forward_batch=forward_batch, ...)
2. Ensure the object passed has a non-None seq_lens tensor (use the real ForwardBatch)
3. In tests, build a stub with a real seq_lens tensor attribute

Example fix

# before
out = algo.retrieve_topk(queries, top_k=64)
# after
out = algo.retrieve_topk(queries, top_k=64, forward_batch=forward_batch)

fb = kwargs.get("forward_batch")
if fb is None or getattr(fb, "seq_lens", None) is None:
    raise ValueError("forward_batch with seq_lens required")

def has_seq_lens(fb) -> bool:
    return fb is not None and getattr(fb, "seq_lens", None) is not None

When it happens

Trigger: Calling retrieve_topk(queries, ...) without kwargs['forward_batch'], or passing a forward_batch object that lacks a seq_lens attribute (e.g. a mocked or partially-built batch).

Common situations: Integrating a sparse-attention retrieval algorithm in a custom model forward that forgot to forward the batch; unit tests with stub forward_batch objects; calling retrieval during a pre-forward hook before seq_lens is populated.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Required: indexer, forward_batch, x, q_lora, positions
• Quest query hidden size {hidden} not divisible by head_dim {
• Unsupported query shape for Quest: {queries.shape}
• Query heads {q_heads} not divisible by KV heads {kv_heads}
• Unknown sparse algorithm: {algorithm_name}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6faafd236a932ffc. Report an issue: GitHub.