sgl-project/sglang · error · RuntimeError

kv-canary: forward_batch.batch_size={bs} exceeds pre-allocat

Error message

kv-canary: forward_batch.batch_size={bs} exceeds pre-allocated write_req_capacity={self._write_req_capacity}; raise --cuda-graph-max-bs or check CanaryLaunchCapacities.from_args

What it means

The kv-canary SingleForwardManager pre-allocates fixed-capacity buffers sized from launch capacities (CanaryLaunchCapacities.from_args). If the runtime forward_batch.batch_size exceeds _write_req_capacity, the pre-allocated write buffers would overflow, so it raises before any kernel launch.

Source

Thrown at python/sglang/srt/kv_canary/single_forward_manager/manager.py:133

        )

    @property
    def phase_checker(self) -> SimplePhaseChecker:
        return self._phase_checker

    def pre_ops_outside_graph(
        self, *, maybe_inaccurate_forward_batch: ForwardBatch
    ) -> None:
        self._phase_checker.update(
            expect_phase=_SingleForwardPhase.IDLE,
            next_phase=_SingleForwardPhase.AFTER_PRE_OUT,
            caller_name="SingleForwardManager.pre_ops_outside_graph",
        )

        bs = int(maybe_inaccurate_forward_batch.batch_size)
        num_tokens = int(maybe_inaccurate_forward_batch.positions.shape[0])
        if bs > self._write_req_capacity:
            raise RuntimeError(
                f"kv-canary: forward_batch.batch_size={bs} exceeds pre-allocated "
                f"write_req_capacity={self._write_req_capacity}; raise --cuda-graph-max-bs "
                f"or check CanaryLaunchCapacities.from_args"
            )
        if num_tokens > self._write_entry_capacity:
            raise RuntimeError(
                f"kv-canary: forward_batch token count={num_tokens} exceeds pre-allocated "
                f"write_entry_capacity={self._write_entry_capacity}; raise "
                f"--chunked-prefill-size / --max-prefill-tokens or check "
                f"CanaryLaunchCapacities.from_args"
            )

        if self._config.enable_verify_token_assert:
            populate_req_to_expected_token_ids(
                forward_batch=maybe_inaccurate_forward_batch,
                req_to_verify_expected_tokens=self._device_state.req_to_verify_expected_tokens,
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Raise --cuda-graph-max-bs to cover the maximum expected batch size
2. Verify CanaryLaunchCapacities.from_args is being computed from the same server args as the live runner
3. Cap concurrency / max-running-requests to stay under the capacity

Example fix

# before
--cuda-graph-max-bs 8
# after
--cuda-graph-max-bs 64

if forward_batch.batch_size > sfm._write_req_capacity:
    raise SystemExit("raise --cuda-graph-max-bs before running kv-canary forward")

When it happens

Trigger: pre_ops_outside_graph receives a forward_batch whose batch_size exceeds the capacity derived from --cuda-graph-max-bs (write_req_capacity) — e.g. a decode batch larger than the configured cuda-graph max bs.

Common situations: Running kv-canary instrumentation with a low --cuda-graph-max-bs while load spikes push running batch size beyond it; capacity args out of sync with actual serving limits.

Related errors

• kv-canary: forward_batch token count={num_tokens} exceeds pr
• kv-canary: offsets kernel bs must be in [0, {_PLAN_BS_BLOCK_
• kv-canary: write_req_capacity must be non-negative, got {wri
• kv-canary: verify_capacity must be non-negative, got {verify
• kv-canary: bs={bs} exceeds write_req_capacity={write_req_cap

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a878678a67defd16. Report an issue: GitHub.