{"record":{"id":"af30e8f19a45b8a1","repo":"sgl-project/sglang","slug":"the-input-size-is-not-aligned-with-the-quantized-w","errorCode":null,"errorMessage":"The input size is not aligned with the quantized weight shape.","messagePattern":"The input size is not aligned with the quantized weight shape\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py","lineNumber":92,"sourceCode":"    def __init__(self, quant_config: BitsAndBytesConfig):\n        require_bitsandbytes()\n        self.quant_config = quant_config\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        output_size: int,\n        params_dtype: torch.dtype,\n        **extra_weight_attrs,\n    ) -> None:\n        quant_ratio = calculate_quant_ratio(params_dtype)\n        output_size_per_partition = sum(output_partition_sizes)\n        total_size = input_size_per_partition * output_size_per_partition\n        if total_size % quant_ratio != 0:\n            raise ValueError(\n                \"The input size is not aligned with the quantized weight shape.\"\n            )\n\n        qweight = nn.Parameter(\n            torch.empty(total_size // quant_ratio, 1, dtype=torch.uint8),\n            requires_grad=False,\n        )\n        set_weight_attrs(\n            qweight,\n            {\n                \"input_dim\": 0,\n                \"output_dim\": 0,\n                \"pack_factor\": quant_ratio,\n                \"use_bitsandbytes_4bit\": True,\n                \"bnb_full_shape\": (output_size, input_size),\n                \"bnb_local_shape\": (\n                    output_size_per_partition,\n                    input_size_per_partition,","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/quantization/bitsandbytes.py#L74-L110","documentation":"bitsandbytes packs multiple elements per byte according to a dtype-dependent quant ratio; the total weight size (input_size_per_partition * output_size_per_partition) must be divisible by that ratio. If not, the packed uint8 buffer cannot be formed correctly.","triggerScenarios":"create_weights for a bnb 4-bit layer where in_features * out_features is not divisible by calculate_quant_ratio(params_dtype) — e.g. odd hidden sizes under tensor parallelism producing non-aligned partition sizes.","commonSituations":"TP sharding splitting a dimension into a size that breaks packing alignment; models with unusual (prime/odd) hidden dims quantized to 4-bit.","solutions":["Choose a tensor-parallel size that divides the dimension evenly and keeps total size aligned","Use a different quantization method for this model shape","Pad/adjust the layer shape if you control the model definition"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"from ...bitsandbytes import calculate_quant_ratio\nif (input_size_per_partition * sum(output_partition_sizes)) % calculate_quant_ratio(params_dtype) != 0:\n    raise SystemExit(\"layer size not aligned for bnb packing\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Choose TP sizes that keep partition sizes aligned","Test quantized loading of unusual hidden dims before deployment"],"tags":["quantization","bitsandbytes","shape-mismatch","alignment"],"backgroundTag":"weight-shape-alignment-error","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}