{"record":{"id":"95110aedf8dfa6f8","repo":"sgl-project/sglang","slug":"num-tokens-num-tokens-exceeds-num-max-dispatch","errorCode":null,"errorMessage":"num_tokens ({num_tokens}) exceeds num_max_dispatch_tokens_per_rank ({self.num_max_dispatch_tokens_per_rank}); raise SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK or lower the per-rank decode batch / chunked-prefill size.","messagePattern":"num_tokens \\((.+?)\\) exceeds num_max_dispatch_tokens_per_rank \\((.+?)\\); raise SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK or lower the per-rank decode batch / chunked-prefill size\\.","errorType":"exception","errorClass":"AssertionError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/layers/moe/token_dispatcher/pplx.py","lineNumber":289,"sourceCode":"        )\n\n        x_q, x_s = sglang_per_token_group_quant_fp8(\n            hidden_states,\n            group_size=_FP8_BLOCK_SIZE,\n        )\n        # pplx expects float32 scales.\n        return x_q, x_s.to(torch.float32)\n\n    def dispatch_a(\n        self,\n        hidden_states: torch.Tensor,\n        topk_output: TopKOutput,\n    ):\n        topk_weights, topk_ids = topk_output.topk_weights, topk_output.topk_ids\n        ata = self._get_all_to_all()\n\n        num_tokens = hidden_states.shape[0]\n        assert num_tokens <= self.num_max_dispatch_tokens_per_rank, (\n            f\"num_tokens ({num_tokens}) exceeds num_max_dispatch_tokens_per_rank \"\n            f\"({self.num_max_dispatch_tokens_per_rank}); raise \"\n            f\"SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK or lower the per-rank \"\n            f\"decode batch / chunked-prefill size.\"\n        )\n        num_dp_groups = get_parallel().attn_dp_size\n        max_batch_tokens = self.num_max_dispatch_tokens_per_rank * num_dp_groups\n        device = hidden_states.device\n\n        dp_x, dp_x_scale = self._quantize(hidden_states)\n        out_expert_num_tokens = torch.zeros(\n            self.num_local_experts, dtype=torch.int32, device=device\n        )\n        out_expert_x = torch.zeros(\n            (self.num_local_experts, max_batch_tokens, self.hidden_size),\n            dtype=dp_x.dtype,\n            device=device,\n        )","sourceCodeStart":271,"sourceCodeEnd":307,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/moe/token_dispatcher/pplx.py#L271-L307","documentation":"Raised by the PPLX (DeepSeek-style) all-to-all MoE token dispatcher when the number of tokens on this rank (hidden_states batch dim) exceeds the pre-allocated num_max_dispatch_tokens_per_rank buffer capacity. The all-to-all communication buffers are sized statically, so any batch larger than that capacity cannot be dispatched. The message tells you to either raise the env var SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK or shrink per-rank batch/chunked-prefill sizes.","triggerScenarios":"Calling dispatch_a (during MoE forward with PPLX token dispatcher / all-to-all attention) with hidden_states.shape[0] > num_max_dispatch_tokens_per_rank, i.e. large chunked-prefill chunks or a large decode batch on one rank, typically when running DeepSeek/Kimi-style models with EP.","commonSituations":"Raising --chunked-prefill-size or --max-running-requests (or DP/TP shrinking per-rank capacity) without raising SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK; long-prompt prefill chunks exceeding the default buffer.","solutions":["Raise the env var, e.g. SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16384 before launching the server","Lower --chunked-prefill-size so per-rank chunks fit the current capacity","Lower --max-running-requests / decode batch size per rank","Check DP/TP/EP parallelism layout: with attention-DP the per-rank token count is the local shard, so resize accordingly"],"exampleFix":"# before\npython -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --chunked-prefill-size 16384\n# after\nSGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32768 \\\n  python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --chunked-prefill-size 16384","handlingStrategy":"validation","validationCode":"import os\ncap = int(os.environ.get(\"SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK\", \"8192\"))\nmax_per_rank_tokens = max(chunked_prefill_size // attn_dp_size, max_running_requests)\nassert max_per_rank_tokens <= cap, f\"raise SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK to >= {max_per_rank_tokens}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK >= max(chunked_prefill_size, max_running_requests) whenever you change batching args","Remember per-rank counts shrink with attention-DP, so recalc after changing -dp","Keep a launch script that derives the env var from server args instead of hardcoding"],"tags":["moe","all-to-all","token-dispatcher","pplx","batch-size","capacity"],"backgroundTag":"buffer-capacity-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}