sgl-project/sglang · error · ValueError

total_verify_tokens {total_verify_tokens} exceeds graph_num_

Error message

total_verify_tokens {total_verify_tokens} exceeds graph_num_tokens {graph_num_tokens}

What it means

The layout's total_verify_tokens must fit within graph_num_tokens (the CUDA-graph capture bucket size). If the ragged batch needs more tokens than the captured graph holds, execution would read/write out of the graph's buffers, so __post_init__ rejects it.

Source

Thrown at python/sglang/srt/speculative/ragged_verify.py:83

        if not self.verify_lens_cpu:
            raise ValueError("RaggedVerifyLayout requires at least one request")
        if min(self.verify_lens_cpu) < 1:
            raise ValueError(
                f"every request must verify the anchor (verify_len >= 1), got "
                f"{self.verify_lens_cpu}"
            )
        if self.cap is not None and max(self.verify_lens_cpu) > self.cap:
            raise ValueError(
                f"capped layout has a row exceeding cap={self.cap}: "
                f"{self.verify_lens_cpu}"
            )
        if self.total_verify_tokens != sum(self.verify_lens_cpu):
            raise ValueError(
                f"total_verify_tokens {self.total_verify_tokens} != "
                f"sum(verify_lens_cpu) {sum(self.verify_lens_cpu)}"
            )
        if not (self.total_verify_tokens <= self.graph_num_tokens):
            raise ValueError(
                f"total_verify_tokens {self.total_verify_tokens} exceeds "
                f"graph_num_tokens {self.graph_num_tokens}"
            )

    @property
    def bs(self) -> int:
        return int(self.verify_lens.shape[0])

    @classmethod
    def _assemble_device(
        cls,
        *,
        verify_lens: torch.Tensor,
        graph_num_tokens: int,
        verify_lens_cpu: Optional[list[int]] = None,
        total_verify_tokens: Optional[int] = None,
        cap: Optional[int] = None,
    ) -> RaggedVerifyLayout:

View on GitHub (pinned to 0132848349)

Solutions

Select a larger capture bucket so total_verify_tokens <= graph_num_tokens
Split the batch across multiple verify forwards
Re-capture ragged verify graphs with a bigger max token budget (cuda_graph_max_bs / token settings)

Example fix

// before
layout = RaggedVerifyLayout(verify_lens_cpu=lens, total_verify_tokens=sum(lens), graph_num_tokens=128)  # sum(lens)=200
// after
layout = RaggedVerifyLayout(verify_lens_cpu=lens, total_verify_tokens=sum(lens), graph_num_tokens=256)

Defensive patterns

Strategy: validation

Validate before calling

if sum(verify_lens) > graph_num_tokens:
    graph_num_tokens = next_bucket(sum(verify_lens))  # or split the batch

Type guard

def fits_graph(lens: list[int], graph_num_tokens: int) -> bool:
    return sum(lens) <= graph_num_tokens

Prevention

Choose capture buckets >= worst-case total verify tokens (bs * (num_draft_tokens+1))
Re-capture graphs after raising batch size or draft length

When it happens

Trigger: Batching more requests (or longer verify rows) than the largest captured graph bucket, e.g. total=512 tokens against graph_num_tokens=256.

Common situations: Increasing batch size or num_draft_tokens without re-capturing larger ragged verify graphs; misaligned bucket selection logic.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/19f1f64d05ad20b4. Report an issue: GitHub.