{"record":{"id":"77b2095aa6da2cf8","repo":"sgl-project/sglang","slug":"the-output-size-of-gate-s-and-up-s-weight-inter-77b209","errorCode":null,"errorMessage":"The output_size of gate's and up's weight = {intermediate_size_per_partition} is not divisible by weight quantization block_n = {block_n}.","messagePattern":"The output_size of gate's and up's weight = (.+?) is not divisible by weight quantization block_n = (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8_moe.py","lineNumber":112,"sourceCode":"    ):\n        from sglang.srt.layers.moe.fused_moe_triton import FusedMoeWeightScaleSupported\n\n        params_dtype = torch.float8_e4m3fn\n\n        if self.block_quant:\n            assert self.weight_block_size is not None\n            layer.weight_block_size = self.weight_block_size\n            tp_size = get_parallel().tp_size\n            block_n, block_k = (\n                self.weight_block_size[0],\n                self.weight_block_size[1],\n            )\n            # NOTE: To ensure proper alignment of the block-wise quantization\n            # scales, the output_size of the weights for both the gate and up\n            # layers must be divisible by block_n.\n            # Required by column parallel or enabling merged weights\n            if intermediate_size_per_partition % block_n != 0:\n                raise ValueError(\n                    f\"The output_size of gate's and up's weight = \"\n                    f\"{intermediate_size_per_partition} is not divisible by \"\n                    f\"weight quantization block_n = {block_n}.\"\n                )\n            if tp_size > 1 and intermediate_size_per_partition % block_k != 0:\n                # Required by row parallel\n                raise ValueError(\n                    f\"The input_size of down's weight = \"\n                    f\"{intermediate_size_per_partition} is not divisible by \"\n                    f\"weight quantization block_k = {block_k}.\"\n                )\n\n        w13_up_dim, w2_down_dim, weight_padded = get_moe_weight_sizes(\n            intermediate_size_per_partition,\n            is_aiter_moe=_use_aiter,\n            is_concat=True,\n            is_packed=False,\n        )","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8_moe.py#L94-L130","documentation":"When using block-quantized (grouped) FP8 MoE weights, the intermediate size per partition must be divisible by the weight block_n so scale tensors stay aligned after column-parallel sharding or merged gate/up projections. This raises when intermediate_size_per_partition % block_n != 0.","triggerScenarios":"Loading a block-quantized FP8 MoE model where (2 * intermediate_size / tp_size) is not divisible by block_n (e.g. block_n=128 with an odd intermediate size or a TP degree that shards it into non-multiples).","commonSituations":"DeepSeek-style FP8 block-quant models launched with an unusual --tp size; models with intermediate_size not a multiple of the quant block size.","solutions":["Change tensor-parallel size so intermediate_size_per_partition is divisible by block_n","Use a checkpoint whose block size divides the intermediate size","Disable TP for this model if its geometry can't be sharded evenly"],"exampleFix":"# before: intermediate_size=5504, block_n=128, --tp 8\n# after: choose --tp 4 so 2*5504/4 is divisible by 128, or use a compatible checkpoint","handlingStrategy":"validation","validationCode":"inter = model_config.intermediate_size; tp = 8; block_n = cfg[\"quantization_config\"][\"weights\"].get(\"block_size\", [128,128])[1]\nassert (2*inter//tp) % block_n == 0, \"intermediate size per partition not divisible by block_n\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Compute shard divisibility for candidate TP sizes before launching","Prefer TP sizes that are powers of two dividing intermediate_size"],"tags":["quantization","fp8","moe","block-quantization","tensor-parallel","shape-mismatch"],"backgroundTag":"tensor-parallel-shape-misaligned","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}