sgl-project/sglang · critical · AssertionError

num_tokens ({num_tokens}) exceeds num_max_dispatch_tokens_pe

Error message

num_tokens ({num_tokens}) exceeds num_max_dispatch_tokens_per_rank ({self.num_max_dispatch_tokens_per_rank}); raise SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK or lower the per-rank decode batch / chunked-prefill size.

What it means

Raised by the PPLX (DeepSeek-style) all-to-all MoE token dispatcher when the number of tokens on this rank (hidden_states batch dim) exceeds the pre-allocated num_max_dispatch_tokens_per_rank buffer capacity. The all-to-all communication buffers are sized statically, so any batch larger than that capacity cannot be dispatched. The message tells you to either raise the env var SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK or shrink per-rank batch/chunked-prefill sizes.

Source

Thrown at python/sglang/srt/layers/moe/token_dispatcher/pplx.py:289

        )

        x_q, x_s = sglang_per_token_group_quant_fp8(
            hidden_states,
            group_size=_FP8_BLOCK_SIZE,
        )
        # pplx expects float32 scales.
        return x_q, x_s.to(torch.float32)

    def dispatch_a(
        self,
        hidden_states: torch.Tensor,
        topk_output: TopKOutput,
    ):
        topk_weights, topk_ids = topk_output.topk_weights, topk_output.topk_ids
        ata = self._get_all_to_all()

        num_tokens = hidden_states.shape[0]
        assert num_tokens <= self.num_max_dispatch_tokens_per_rank, (
            f"num_tokens ({num_tokens}) exceeds num_max_dispatch_tokens_per_rank "
            f"({self.num_max_dispatch_tokens_per_rank}); raise "
            f"SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK or lower the per-rank "
            f"decode batch / chunked-prefill size."
        )
        num_dp_groups = get_parallel().attn_dp_size
        max_batch_tokens = self.num_max_dispatch_tokens_per_rank * num_dp_groups
        device = hidden_states.device

        dp_x, dp_x_scale = self._quantize(hidden_states)
        out_expert_num_tokens = torch.zeros(
            self.num_local_experts, dtype=torch.int32, device=device
        )
        out_expert_x = torch.zeros(
            (self.num_local_experts, max_batch_tokens, self.hidden_size),
            dtype=dp_x.dtype,
            device=device,
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Raise the env var, e.g. SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16384 before launching the server
2. Lower --chunked-prefill-size so per-rank chunks fit the current capacity
3. Lower --max-running-requests / decode batch size per rank
4. Check DP/TP/EP parallelism layout: with attention-DP the per-rank token count is the local shard, so resize accordingly

Example fix

# before
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --chunked-prefill-size 16384
# after
SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32768 \
  python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --chunked-prefill-size 16384

import os
cap = int(os.environ.get("SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK", "8192"))
max_per_rank_tokens = max(chunked_prefill_size // attn_dp_size, max_running_requests)
assert max_per_rank_tokens <= cap, f"raise SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK to >= {max_per_rank_tokens}"

When it happens

Trigger: Calling dispatch_a (during MoE forward with PPLX token dispatcher / all-to-all attention) with hidden_states.shape[0] > num_max_dispatch_tokens_per_rank, i.e. large chunked-prefill chunks or a large decode batch on one rank, typically when running DeepSeek/Kimi-style models with EP.

Common situations: Raising --chunked-prefill-size or --max-running-requests (or DP/TP shrinking per-rank capacity) without raising SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK; long-prompt prefill chunks exceeding the default buffer.

Related errors

• kv-canary: forward_batch.batch_size={bs} exceeds pre-allocat
• cannot found moe_block_size for shape {valid_shape_m}
• moe_a2a_backend='pplx' only supports low-latency mode; set -
• kv-canary: offsets kernel bs must be in [0, {_PLAN_BS_BLOCK_
• kv-canary: write_req_capacity must be non-negative, got {wri

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/95110aedf8dfa6f8. Report an issue: GitHub.