{"record":{"id":"a878678a67defd16","repo":"sgl-project/sglang","slug":"kv-canary-forward-batch-batch-size-bs-exceeds-p","errorCode":null,"errorMessage":"kv-canary: forward_batch.batch_size={bs} exceeds pre-allocated write_req_capacity={self._write_req_capacity}; raise --cuda-graph-max-bs or check CanaryLaunchCapacities.from_args","messagePattern":"kv-canary: forward_batch\\.batch_size=(.+?) exceeds pre-allocated write_req_capacity=(.+?); raise --cuda-graph-max-bs or check CanaryLaunchCapacities\\.from_args","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/kv_canary/single_forward_manager/manager.py","lineNumber":133,"sourceCode":"        )\n\n    @property\n    def phase_checker(self) -> SimplePhaseChecker:\n        return self._phase_checker\n\n    def pre_ops_outside_graph(\n        self, *, maybe_inaccurate_forward_batch: ForwardBatch\n    ) -> None:\n        self._phase_checker.update(\n            expect_phase=_SingleForwardPhase.IDLE,\n            next_phase=_SingleForwardPhase.AFTER_PRE_OUT,\n            caller_name=\"SingleForwardManager.pre_ops_outside_graph\",\n        )\n\n        bs = int(maybe_inaccurate_forward_batch.batch_size)\n        num_tokens = int(maybe_inaccurate_forward_batch.positions.shape[0])\n        if bs > self._write_req_capacity:\n            raise RuntimeError(\n                f\"kv-canary: forward_batch.batch_size={bs} exceeds pre-allocated \"\n                f\"write_req_capacity={self._write_req_capacity}; raise --cuda-graph-max-bs \"\n                f\"or check CanaryLaunchCapacities.from_args\"\n            )\n        if num_tokens > self._write_entry_capacity:\n            raise RuntimeError(\n                f\"kv-canary: forward_batch token count={num_tokens} exceeds pre-allocated \"\n                f\"write_entry_capacity={self._write_entry_capacity}; raise \"\n                f\"--chunked-prefill-size / --max-prefill-tokens or check \"\n                f\"CanaryLaunchCapacities.from_args\"\n            )\n\n        if self._config.enable_verify_token_assert:\n            populate_req_to_expected_token_ids(\n                forward_batch=maybe_inaccurate_forward_batch,\n                req_to_verify_expected_tokens=self._device_state.req_to_verify_expected_tokens,\n            )\n","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/kv_canary/single_forward_manager/manager.py#L115-L151","documentation":"The kv-canary SingleForwardManager pre-allocates fixed-capacity buffers sized from launch capacities (CanaryLaunchCapacities.from_args). If the runtime forward_batch.batch_size exceeds _write_req_capacity, the pre-allocated write buffers would overflow, so it raises before any kernel launch.","triggerScenarios":"pre_ops_outside_graph receives a forward_batch whose batch_size exceeds the capacity derived from --cuda-graph-max-bs (write_req_capacity) — e.g. a decode batch larger than the configured cuda-graph max bs.","commonSituations":"Running kv-canary instrumentation with a low --cuda-graph-max-bs while load spikes push running batch size beyond it; capacity args out of sync with actual serving limits.","solutions":["Raise --cuda-graph-max-bs to cover the maximum expected batch size","Verify CanaryLaunchCapacities.from_args is being computed from the same server args as the live runner","Cap concurrency / max-running-requests to stay under the capacity"],"exampleFix":"# before\n--cuda-graph-max-bs 8\n# after\n--cuda-graph-max-bs 64","handlingStrategy":"validation","validationCode":"if forward_batch.batch_size > sfm._write_req_capacity:\n    raise SystemExit(\"raise --cuda-graph-max-bs before running kv-canary forward\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive capacities once from the same server_args used to launch the runner","Set --cuda-graph-max-bs >= max running requests in load tests"],"tags":["kv-canary","capacity","cuda-graph-max-bs","batch-size","sglang"],"backgroundTag":"preallocated-buffer-overflow","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}