{"record":{"id":"41ba193305383fc7","repo":"sgl-project/sglang","slug":"kv-canary-forward-batch-token-count-num-tokens","errorCode":null,"errorMessage":"kv-canary: forward_batch token count={num_tokens} exceeds pre-allocated write_entry_capacity={self._write_entry_capacity}; raise --chunked-prefill-size / --max-prefill-tokens or check CanaryLaunchCapacities.from_args","messagePattern":"kv-canary: forward_batch token count=(.+?) exceeds pre-allocated write_entry_capacity=(.+?); raise --chunked-prefill-size / --max-prefill-tokens or check CanaryLaunchCapacities\\.from_args","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/kv_canary/single_forward_manager/manager.py","lineNumber":139,"sourceCode":"    def pre_ops_outside_graph(\n        self, *, maybe_inaccurate_forward_batch: ForwardBatch\n    ) -> None:\n        self._phase_checker.update(\n            expect_phase=_SingleForwardPhase.IDLE,\n            next_phase=_SingleForwardPhase.AFTER_PRE_OUT,\n            caller_name=\"SingleForwardManager.pre_ops_outside_graph\",\n        )\n\n        bs = int(maybe_inaccurate_forward_batch.batch_size)\n        num_tokens = int(maybe_inaccurate_forward_batch.positions.shape[0])\n        if bs > self._write_req_capacity:\n            raise RuntimeError(\n                f\"kv-canary: forward_batch.batch_size={bs} exceeds pre-allocated \"\n                f\"write_req_capacity={self._write_req_capacity}; raise --cuda-graph-max-bs \"\n                f\"or check CanaryLaunchCapacities.from_args\"\n            )\n        if num_tokens > self._write_entry_capacity:\n            raise RuntimeError(\n                f\"kv-canary: forward_batch token count={num_tokens} exceeds pre-allocated \"\n                f\"write_entry_capacity={self._write_entry_capacity}; raise \"\n                f\"--chunked-prefill-size / --max-prefill-tokens or check \"\n                f\"CanaryLaunchCapacities.from_args\"\n            )\n\n        if self._config.enable_verify_token_assert:\n            populate_req_to_expected_token_ids(\n                forward_batch=maybe_inaccurate_forward_batch,\n                req_to_verify_expected_tokens=self._device_state.req_to_verify_expected_tokens,\n            )\n\n    def pre_ops_maybe_inside_graph(\n        self, forward_batch: ForwardBatch\n    ) -> _PreOpsMaybeInsideGraphOutput:\n        self._phase_checker.update(\n            expect_phase=_SingleForwardPhase.AFTER_PRE_OUT,\n            next_phase=_SingleForwardPhase.AFTER_PRE_MAYBE_IN,","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/kv_canary/single_forward_manager/manager.py#L121-L157","documentation":"Companion to the batch-size check: the kv-canary write buffers are sized for a maximum token count per forward (_write_entry_capacity, derived from --chunked-prefill-size / --max-prefill-tokens). A forward_batch with more tokens than that capacity would overflow the pre-allocated entries and is rejected.","triggerScenarios":"pre_ops_outside_graph gets a forward_batch where positions.shape[0] (total tokens, e.g. a chunked prefill chunk) exceeds the entry capacity derived from chunked-prefill-size / max-prefill-tokens.","commonSituations":"Raising --max-prefill-tokens or --chunked-prefill-size at request time, or mixed prefill batches larger than what CanaryLaunchCapacities.from_args computed; multi-token decode with large speculative verify batches.","solutions":["Raise --chunked-prefill-size / --max-prefill-tokens so capacity covers the largest forward batch","Recompute/align CanaryLaunchCapacities.from_args with the live server args","Lower speculative num_draft_tokens if verify token counts blow past capacity"],"exampleFix":"# before\n--chunked-prefill-size 2048\n# after\n--chunked-prefill-size 8192","handlingStrategy":"validation","validationCode":"if forward_batch.positions.shape[0] > sfm._write_entry_capacity:\n    raise SystemExit(\"raise --chunked-prefill-size / --max-prefill-tokens\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep --chunked-prefill-size and --max-prefill-tokens consistent with canary capacities","Bound verify token counts when using large speculative draft lengths"],"tags":["kv-canary","capacity","chunked-prefill","max-prefill-tokens","sglang"],"backgroundTag":"preallocated-buffer-overflow","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}