{"record":{"id":"70334a228c5a0a4e","repo":"sgl-project/sglang","slug":"weight-input-size-per-partition-input-size-per","errorCode":null,"errorMessage":"Weight input_size_per_partition = {input_size_per_partition} is not divisible by weight quantization block_k = {block_k}.","messagePattern":"Weight input_size_per_partition = (.+?) is not divisible by weight quantization block_k = (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/quantization/fp8.py","lineNumber":158,"sourceCode":"        output_partition_sizes: List[int],\n        input_size: int,\n        output_size: int,\n        params_dtype: torch.dtype,\n        **extra_weight_attrs,\n    ):\n        output_size_per_partition = sum(output_partition_sizes)\n        weight_loader = extra_weight_attrs.get(\"weight_loader\")\n\n        tp_size = get_tensor_model_parallel_world_size()\n        if self.block_quant:\n            block_n, block_k = (\n                self.quant_config.weight_block_size[0],\n                self.quant_config.weight_block_size[1],\n            )\n            # Required by row parallel\n            if tp_size > 1 and input_size // input_size_per_partition == tp_size:\n                if input_size_per_partition % block_k != 0:\n                    raise ValueError(\n                        f\"Weight input_size_per_partition = \"\n                        f\"{input_size_per_partition} is not divisible by \"\n                        f\"weight quantization block_k = {block_k}.\"\n                    )\n            # Required by column parallel or enabling merged weights\n            if (\n                tp_size > 1 and output_size // output_size_per_partition == tp_size\n            ) or len(output_partition_sizes) > 1:\n                for output_partition_size in output_partition_sizes:\n                    if output_partition_size % block_n != 0:\n                        raise ValueError(\n                            f\"Weight output_partition_size = \"\n                            f\"{output_partition_size} is not divisible by \"\n                            f\"weight quantization block_n = {block_n}.\"\n                        )\n\n        layer.logical_widths = output_partition_sizes\n        layer.input_size_per_partition = input_size_per_partition","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/quantization/fp8.py#L140-L176","documentation":"Raised in create_weights of the FP8 quantization method when, under tensor parallelism with row-parallel weight sharding, the per-partition input dimension (input_size // tp_size) is not divisible by the FP8 blockwise weight block size (weight_block_size[1], i.e. block_k). Blockwise FP8 quantization requires each shard to align to quantization block boundaries.","triggerScenarios":"Creating FP8 blockwise-quantized weights with tp_size>1 where input_size (in_features) / tp_size leaves a remainder modulo weight_block_size[1] (commonly 128); e.g. in_features=6144 with tp=8 gives 768 which is fine, but in_features=3000 with tp=2 gives 1500, not divisible by 128.","commonSituations":"Choosing a TP degree that doesn't evenly split hidden dims into 128-blocks, loading a model whose intermediate size isn't block-aligned, or overriding --quantization fp8 with --weight-block-size on a model not designed for blockwise FP8.","solutions":["Pick a tensor-parallel size where input_size/tp_size is divisible by weight_block_size[1] (usually 128) — e.g. reduce tp_size until aligned","Check the model config's hidden/intermediate size for blockwise-FP8 compatibility before launching","Use a non-blockwise FP8 scheme (per-tensor scales) if the dimension can't be aligned","Fall back to tp_size=1 for the offending layer group or use a different quantization method"],"exampleFix":"# before\n--tensor-parallel-size 6   # 4608/6=768 ok, but 3000/6=500 not % 128\n# after\n--tensor-parallel-size 2   # ensure input_size/tp % block_k == 0\n# or validate up front:\nassert (hidden // tp) % quant_config.weight_block_size[1] == 0","handlingStrategy":"validation","validationCode":"block_k = quant_config.weight_block_size[1]\nfor layer in linear_layers:\n    if tp_size > 1 and layer.input_size % tp_size == 0:\n        assert (layer.input_size // tp_size) % block_k == 0, f'{layer.input_size}/{tp_size} not % {block_k}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer TP sizes that divide hidden sizes into multiples of 128","Check model config dims against weight_block_size before enabling blockwise FP8","Keep a per-model table of validated TP sizes for blockwise-quantized checkpoints"],"tags":["fp8","quantization","tensor-parallel","block-size"],"backgroundTag":"tensor-parallel-dimension-misaligned","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}