{"record":{"id":"19f1f64d05ad20b4","repo":"sgl-project/sglang","slug":"total-verify-tokens-total-verify-tokens-exceeds","errorCode":null,"errorMessage":"total_verify_tokens {total_verify_tokens} exceeds graph_num_tokens {graph_num_tokens}","messagePattern":"total_verify_tokens (.+?) exceeds graph_num_tokens (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/speculative/ragged_verify.py","lineNumber":83,"sourceCode":"        if not self.verify_lens_cpu:\n            raise ValueError(\"RaggedVerifyLayout requires at least one request\")\n        if min(self.verify_lens_cpu) < 1:\n            raise ValueError(\n                f\"every request must verify the anchor (verify_len >= 1), got \"\n                f\"{self.verify_lens_cpu}\"\n            )\n        if self.cap is not None and max(self.verify_lens_cpu) > self.cap:\n            raise ValueError(\n                f\"capped layout has a row exceeding cap={self.cap}: \"\n                f\"{self.verify_lens_cpu}\"\n            )\n        if self.total_verify_tokens != sum(self.verify_lens_cpu):\n            raise ValueError(\n                f\"total_verify_tokens {self.total_verify_tokens} != \"\n                f\"sum(verify_lens_cpu) {sum(self.verify_lens_cpu)}\"\n            )\n        if not (self.total_verify_tokens <= self.graph_num_tokens):\n            raise ValueError(\n                f\"total_verify_tokens {self.total_verify_tokens} exceeds \"\n                f\"graph_num_tokens {self.graph_num_tokens}\"\n            )\n\n    @property\n    def bs(self) -> int:\n        return int(self.verify_lens.shape[0])\n\n    @classmethod\n    def _assemble_device(\n        cls,\n        *,\n        verify_lens: torch.Tensor,\n        graph_num_tokens: int,\n        verify_lens_cpu: Optional[list[int]] = None,\n        total_verify_tokens: Optional[int] = None,\n        cap: Optional[int] = None,\n    ) -> RaggedVerifyLayout:","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/speculative/ragged_verify.py#L65-L101","documentation":"The layout's total_verify_tokens must fit within graph_num_tokens (the CUDA-graph capture bucket size). If the ragged batch needs more tokens than the captured graph holds, execution would read/write out of the graph's buffers, so __post_init__ rejects it.","triggerScenarios":"Batching more requests (or longer verify rows) than the largest captured graph bucket, e.g. total=512 tokens against graph_num_tokens=256.","commonSituations":"Increasing batch size or num_draft_tokens without re-capturing larger ragged verify graphs; misaligned bucket selection logic.","solutions":["Select a larger capture bucket so total_verify_tokens <= graph_num_tokens","Split the batch across multiple verify forwards","Re-capture ragged verify graphs with a bigger max token budget (cuda_graph_max_bs / token settings)"],"exampleFix":"// before\nlayout = RaggedVerifyLayout(verify_lens_cpu=lens, total_verify_tokens=sum(lens), graph_num_tokens=128)  # sum(lens)=200\n// after\nlayout = RaggedVerifyLayout(verify_lens_cpu=lens, total_verify_tokens=sum(lens), graph_num_tokens=256)","handlingStrategy":"validation","validationCode":"if sum(verify_lens) > graph_num_tokens:\n    graph_num_tokens = next_bucket(sum(verify_lens))  # or split the batch","typeGuard":"def fits_graph(lens: list[int], graph_num_tokens: int) -> bool:\n    return sum(lens) <= graph_num_tokens","tryCatchPattern":null,"preventionTips":["Choose capture buckets >= worst-case total verify tokens (bs * (num_draft_tokens+1))","Re-capture graphs after raising batch size or draft length"],"tags":["speculative-decoding","cuda-graph","validation"],"backgroundTag":"capacity-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}