sgl-project/sglang · error · ValueError

Weight input_size_per_partition = {input_size_per_partition}

Error message

Weight input_size_per_partition = {input_size_per_partition} is not divisible by group_size = {self.quant_config.group_size}.

What it means

When group_size != -1, Marlin stores one scale per group along K, so input_size_per_partition must be divisible by group_size (typically 128). create_weights raises this if the TP-sharded local K cuts through a quant group, leaving scales unaddressable.

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:785

        if output_size_per_partition % self.quant_config.pack_factor != 0:
            raise ValueError(
                f"Weight output_size_per_partition = "
                f"{output_size_per_partition} is not divisible by "
                f"pack_factor = {self.quant_config.pack_factor}."
            )

        # Validate input_size_per_partition
        if input_size_per_partition % self.quant_config.min_k_threads != 0:
            raise ValueError(
                f"Weight input_size_per_partition = "
                f"{input_size_per_partition} is not divisible by "
                f"min_k_threads = {self.quant_config.min_k_threads}."
            )
        if (
            self.quant_config.group_size != -1
            and input_size_per_partition % self.quant_config.group_size != 0
        ):
            raise ValueError(
                f"Weight input_size_per_partition = "
                f"{input_size_per_partition} is not divisible by "
                f"group_size = {self.quant_config.group_size}."
            )

        # Check that we have at least 4 tiles horizontally in the shard
        num_tiles_per_perm = self.quant_config.perm_len // (
            self.quant_config.tile_size**2
        )
        if output_size_per_partition % num_tiles_per_perm != 0:
            raise ValueError("Each permutation group must reside on the same gpu")

        # Quantized 4Bit weights packed into Int32.
        qweight = PackedvLLMParameter(
            data=torch.empty(
                input_size_per_partition // self.quant_config.tile_size,
                output_size_per_partition
                * self.quant_config.tile_size

View on GitHub (pinned to 0132848349)

Solutions

Choose TP in {1,2,4,8,...} so local K is a multiple of group_size (128)
Re-quantize channel-wise (group_size -1) if TP flexibility matters more than accuracy
Verify quantization_config.group_size divides hidden_size / TP exactly

Example fix

# before
--tensor-parallel-size 3
# after
--tensor-parallel-size 4  # local K = hidden/4 stays multiple of 128

Defensive patterns

Strategy: validation

Validate before calling

gs = cfg.quantization_config.get("group_size", -1)
if gs != -1:
    assert (hidden_size // tp) % gs == 0, "local K cuts a quant group"

Prevention

Keep TP a power of two so local K stays a multiple of group_size 128
Or use channel-wise (-1) quantization when TP flexibility is needed

When it happens

Trigger: Local K after sharding not divisible by group_size 128, e.g. hidden 5120 with TP=3 giving 1706.67-group misalignment; QKV fused layers whose per-partition K mixes dimensions.

Common situations: Non-power-of-two TP; checkpoints quantized with group_size 128 but served with TP degrees that don't divide the K dim into group multiples.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4b2251085bcf955a. Report an issue: GitHub.