{"record":{"id":"81e7ef50eb196c32","repo":"sgl-project/sglang","slug":"the-output-size-of-gate-s-and-up-s-weight-inter-81e7ef","errorCode":null,"errorMessage":"The output_size of gate's and up's weight = {intermediate_size_per_partition} is not divisible by weight quantization block_n = {block_n}.","messagePattern":"The output_size of gate's and up's weight = (.+?) is not divisible by weight quantization block_n = (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/fp8.py","lineNumber":1170,"sourceCode":"        w13_up_dim, w2_up_dim, weight_padded = get_moe_weight_sizes(\n            intermediate_size_per_partition,\n            is_aiter_moe=_use_aiter,\n            is_concat=layer.moe_runner_config.is_gated,\n            is_packed=False,\n        )\n\n        if block_quant:\n            block_n, block_k = (\n                quant_config.weight_block_size[0],\n                quant_config.weight_block_size[1],\n            )\n\n            padding_size = get_moe_padding_size(_use_aiter)\n            if not (_use_aiter and padding_size == block_n == block_k):\n                # NOTE(HandH1998): To ensure proper alignment of the block-wise quantization scales, the output_size of the weights for both the gate and up layers must be divisible by block_n.\n                # Required by column parallel or enabling merged weights\n                if intermediate_size_per_partition % block_n != 0:\n                    raise ValueError(\n                        f\"The output_size of gate's and up's weight = \"\n                        f\"{intermediate_size_per_partition} is not divisible by \"\n                        f\"weight quantization block_n = {block_n}.\"\n                    )\n                if tp_size > 1:\n                    # Required by row parallel\n                    if intermediate_size_per_partition % block_k != 0:\n                        raise ValueError(\n                            f\"The input_size of down's weight = \"\n                            f\"{intermediate_size_per_partition} is not divisible by \"\n                            f\"weight quantization block_k = {block_k}.\"\n                        )\n\n        # WEIGHTS\n        if is_fp4_expert:\n            w13_weight = torch.nn.Parameter(\n                torch.empty(\n                    num_experts,","sourceCodeStart":1152,"sourceCodeEnd":1188,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/fp8.py#L1152-L1188","documentation":"For block-quant FP8 MoE layers (unless on ROCm AIter with matching padding), the per-partition intermediate size (gate/up output size) must be divisible by block_n so FP8 block scale rows align. create_fp8_moe_weight_ raises during weight creation, i.e. at model init.","triggerScenarios":"Creating FP8 MoE weights with intermediate_size_per_partition = intermediate_size / tp not divisible by block_n (128) — e.g. moe_intermediate_size 1408 with TP=2 giving 704, not a multiple of 128.","commonSituations":"Running FP8 MoE models (DeepSeek, Qwen-MoE, Mixtral-FP8) with TP sizes that shard intermediate size into non-128 multiples; models with small moe_intermediate_size.","solutions":["Pick a TP size where moe_intermediate_size/tp is a multiple of 128 (often TP=1,2,4,8)","Check the model's moe_intermediate_size and compute divisibility before launching","Use a non-block-quant checkpoint or an AIter path on ROCm gfx95 if applicable"],"exampleFix":"# before\n--tp 6  # 1536/6 = 256 ok, but e.g. 1408/6 not multiple of 128\n# after\n--tp 2  # intermediate_size_per_partition % 128 == 0","handlingStrategy":"validation","validationCode":"inter = model_config.moe_intermediate_size; block_n = 128; tp = args.tp\nassert tp == 1 or (inter // tp) % block_n == 0, f\"moe intermediate/tp={inter//tp} not divisible by {block_n}\"","typeGuard":"def moe_tp_ok(inter_size: int, tp: int, block: int = 128) -> bool:\n    return tp == 1 or (inter_size // tp) % block == 0","tryCatchPattern":null,"preventionTips":["Pre-validate moe_intermediate_size/tp against block size in launch scripts","Prefer standard TP sizes on FP8 MoE models"],"tags":["quantization","fp8","moe","tensor-parallel","shape-validation"],"backgroundTag":"tensor-parallel-shape-misaligned","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}