{"record":{"id":"4004352387ff15a3","repo":"sgl-project/sglang","slug":"mega-moe-num-tokens-num-tokens-exceeds-sglang-o","errorCode":null,"errorMessage":"mega MoE: num_tokens={num_tokens} exceeds SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK={num_max_tokens_per_rank}; K3 has no non-mega fallback — raise the env var to cover the per-rank rows","messagePattern":"mega MoE: num_tokens=(.+?) exceeds SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=(.+?); K3 has no non-mega fallback — raise the env var to cover the per-rank rows","errorType":"exception","errorClass":"AssertionError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/models/kimi_k3.py","lineNumber":785,"sourceCode":"        backend (combine returns fully-summed rows; `_reduce_latent` then only\n        applies the norm).\"\"\"\n        import deep_gemm\n\n        from sglang.kernels.ops.attention.dsv4 import mega_moe_pre_dispatch\n        from sglang.srt.distributed.parallel_state import get_moe_ep_group\n        from sglang.srt.environ import envs\n        from sglang.srt.layers.moe.mega_moe import _get_mega_moe_symm_buffer\n\n        # In SP-MoE mode (KimiK3DecoderLayer reduce-scatters the o_proj\n        # output) the incoming rows are already this rank's token shard, so\n        # the fused a2a below dispatches each token exactly once. On the\n        # non-scattered fallback path the rows are the full batch (redundant\n        # across ranks but correct).\n        num_tokens = routed_input.shape[0]\n        num_max_tokens_per_rank = (\n            envs.SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK.get()\n        )\n        assert num_tokens <= num_max_tokens_per_rank, (\n            f\"mega MoE: num_tokens={num_tokens} exceeds \"\n            f\"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=\"\n            f\"{num_max_tokens_per_rank}; K3 has no non-mega fallback — raise \"\n            f\"the env var to cover the per-rank rows\"\n        )\n        buf = _get_mega_moe_symm_buffer(\n            get_moe_ep_group().device_group,\n            num_experts=self.experts.num_experts,\n            num_max_tokens_per_rank=num_max_tokens_per_rank,\n            num_topk=self._mega_top_k,\n            hidden=self.moe_hidden_size,\n            intermediate_hidden=self._mega_intermediate_size,\n        )\n\n        if num_tokens > 0:\n            topk_ids_in = topk_output.topk_ids.to(torch.int32)\n            topk_weights_in = topk_output.topk_weights.to(torch.float32)\n        else:","sourceCodeStart":767,"sourceCodeEnd":803,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/kimi_k3.py#L767-L803","documentation":"Kimi K3's mega-MoE DeepGEMM path allocates symmetric buffers sized by SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK and asserts routed_input rows fit. Unlike other models, K3 has no non-mega fallback, so overflow is a hard failure rather than a graceful path switch.","triggerScenarios":"Running Kimi K3 with the mega MoE (OPT DeepGEMM) path when routed_input.shape[0] (per-rank scattered rows, or full batch rows in the non-scattered path) exceeds the env-var limit — e.g. big prefill chunks or a whole-batch fallback batch.","commonSituations":"Serving Kimi K3 with large --chunked-prefill-size or high concurrency; the default mega-MoE token cap being smaller than the actual per-rank workload.","solutions":["Raise SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK above the per-rank row count (prefill chunk or full batch, whichever path runs)","Lower --chunked-prefill-size / batch size to fit the current cap","If memory-bound, reduce parallelism per node or free memory before raising the cap since buffers scale with it"],"exampleFix":"# before\npython -m sglang.launch_server --model moonshot-ai/Kimi-K3 --chunked-prefill-size 32768\n# after\nSGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=65536 \\\n  python -m sglang.launch_server --model moonshot-ai/Kimi-K3 --chunked-prefill-size 32768","handlingStrategy":"validation","validationCode":"cap = int(os.environ.get(\"SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK\", \"0\") or 0)\nneeded = max(chunked_prefill_size, batch_rows)\nassert cap >= needed, f\"set SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK>={needed}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["For Kimi K3, always set the mega-MoE cap to at least the largest per-rank batch (there is no fallback)","Budget GPU memory: symmetric buffers scale with the cap","Re-verify after changing chunked-prefill or DP settings"],"tags":["moe","deepgemm","kimi-k3","mega-moe","buffer-capacity","env-var"],"backgroundTag":"buffer-capacity-exceeded","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}