sgl-project/sglang · error · RuntimeError

kv-canary: forward_batch token count={num_tokens} exceeds pr

Error message

kv-canary: forward_batch token count={num_tokens} exceeds pre-allocated write_entry_capacity={self._write_entry_capacity}; raise --chunked-prefill-size / --max-prefill-tokens or check CanaryLaunchCapacities.from_args

What it means

Companion to the batch-size check: the kv-canary write buffers are sized for a maximum token count per forward (_write_entry_capacity, derived from --chunked-prefill-size / --max-prefill-tokens). A forward_batch with more tokens than that capacity would overflow the pre-allocated entries and is rejected.

Source

Thrown at python/sglang/srt/kv_canary/single_forward_manager/manager.py:139

    def pre_ops_outside_graph(
        self, *, maybe_inaccurate_forward_batch: ForwardBatch
    ) -> None:
        self._phase_checker.update(
            expect_phase=_SingleForwardPhase.IDLE,
            next_phase=_SingleForwardPhase.AFTER_PRE_OUT,
            caller_name="SingleForwardManager.pre_ops_outside_graph",
        )

        bs = int(maybe_inaccurate_forward_batch.batch_size)
        num_tokens = int(maybe_inaccurate_forward_batch.positions.shape[0])
        if bs > self._write_req_capacity:
            raise RuntimeError(
                f"kv-canary: forward_batch.batch_size={bs} exceeds pre-allocated "
                f"write_req_capacity={self._write_req_capacity}; raise --cuda-graph-max-bs "
                f"or check CanaryLaunchCapacities.from_args"
            )
        if num_tokens > self._write_entry_capacity:
            raise RuntimeError(
                f"kv-canary: forward_batch token count={num_tokens} exceeds pre-allocated "
                f"write_entry_capacity={self._write_entry_capacity}; raise "
                f"--chunked-prefill-size / --max-prefill-tokens or check "
                f"CanaryLaunchCapacities.from_args"
            )

        if self._config.enable_verify_token_assert:
            populate_req_to_expected_token_ids(
                forward_batch=maybe_inaccurate_forward_batch,
                req_to_verify_expected_tokens=self._device_state.req_to_verify_expected_tokens,
            )

    def pre_ops_maybe_inside_graph(
        self, forward_batch: ForwardBatch
    ) -> _PreOpsMaybeInsideGraphOutput:
        self._phase_checker.update(
            expect_phase=_SingleForwardPhase.AFTER_PRE_OUT,
            next_phase=_SingleForwardPhase.AFTER_PRE_MAYBE_IN,

View on GitHub (pinned to 0132848349)

Solutions

1. Raise --chunked-prefill-size / --max-prefill-tokens so capacity covers the largest forward batch
2. Recompute/align CanaryLaunchCapacities.from_args with the live server args
3. Lower speculative num_draft_tokens if verify token counts blow past capacity

Example fix

# before
--chunked-prefill-size 2048
# after
--chunked-prefill-size 8192

if forward_batch.positions.shape[0] > sfm._write_entry_capacity:
    raise SystemExit("raise --chunked-prefill-size / --max-prefill-tokens")

When it happens

Trigger: pre_ops_outside_graph gets a forward_batch where positions.shape[0] (total tokens, e.g. a chunked prefill chunk) exceeds the entry capacity derived from chunked-prefill-size / max-prefill-tokens.

Common situations: Raising --max-prefill-tokens or --chunked-prefill-size at request time, or mixed prefill batches larger than what CanaryLaunchCapacities.from_args computed; multi-token decode with large speculative verify batches.

Related errors

• kv-canary: forward_batch.batch_size={bs} exceeds pre-allocat
• kv-canary: write_req_capacity must be non-negative, got {wri
• kv-canary: verify_capacity must be non-negative, got {verify
• kv-canary: bs={bs} exceeds write_req_capacity={write_req_cap
• kv-canary: VerifyPlan verify_capacity must be positive, got

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/41ba193305383fc7. Report an issue: GitHub.