sgl-project/sglang · error · ValueError

Weight input_size_per_partition = {input_size_per_partition}

Error message

Weight input_size_per_partition = {input_size_per_partition} is not divisible by min_k_threads = {self.quant_config.min_k_threads}.

What it means

The input (K) dimension per partition must be divisible by the Marlin config's min_k_threads because the kernel processes K in fixed thread-group tiles. create_weights raises this when the TP-sharded local K is misaligned.

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:776

        # Validate output_size_per_partition
        output_size_per_partition = sum(output_partition_sizes)
        if output_size_per_partition % self.quant_config.min_n_threads != 0:
            raise ValueError(
                f"Weight output_size_per_partition = "
                f"{output_size_per_partition} is not divisible by "
                f"min_n_threads = {self.quant_config.min_n_threads}."
            )
        if output_size_per_partition % self.quant_config.pack_factor != 0:
            raise ValueError(
                f"Weight output_size_per_partition = "
                f"{output_size_per_partition} is not divisible by "
                f"pack_factor = {self.quant_config.pack_factor}."
            )

        # Validate input_size_per_partition
        if input_size_per_partition % self.quant_config.min_k_threads != 0:
            raise ValueError(
                f"Weight input_size_per_partition = "
                f"{input_size_per_partition} is not divisible by "
                f"min_k_threads = {self.quant_config.min_k_threads}."
            )
        if (
            self.quant_config.group_size != -1
            and input_size_per_partition % self.quant_config.group_size != 0
        ):
            raise ValueError(
                f"Weight input_size_per_partition = "
                f"{input_size_per_partition} is not divisible by "
                f"group_size = {self.quant_config.group_size}."
            )

        # Check that we have at least 4 tiles horizontally in the shard
        num_tiles_per_perm = self.quant_config.perm_len // (
            self.quant_config.tile_size**2
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce tensor_parallel_size to a power-of-two divisor of the input dimension
2. Use --quantization gptq fallback for exotic dims
3. Confirm hidden_size / TP >= and divisible by min_k_threads before launch

Example fix

# before
--tensor-parallel-size 6  # 4096/6 not divisible by min_k_threads
# after
--tensor-parallel-size 2

min_k = 256  # GPTQ Marlin min_k_threads
assert (hidden_size // tp) % min_k == 0, "local K breaks Marlin K tiling"

When it happens

Trigger: hidden_size or local K after TP/MoE sharding not divisible by min_k_threads (commonly 256 for Marlin); embedding/lm-head layers with unusual dims.

Common situations: Small models (tiny hidden sizes) with Marlin checkpoints; TP degrees like 3 or 6 breaking power-of-two alignment.

Related errors

• Weight output_size_per_partition = {output_size_per_partitio
• Weight output_size_per_partition = {output_size_per_partitio
• Weight input_size_per_partition = {input_size_per_partition}
• num_bits must be 4 or 8, got {}
• Currently, only group size 128 and -1 (channelwise) is suppo

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5561dcfa42e5eedd. Report an issue: GitHub.