sgl-project/sglang · error · ValueError

Weight output_size_per_partition = {output_size_per_partitio

Error message

Weight output_size_per_partition = {output_size_per_partition} is not divisible by min_n_threads = {self.quant_config.min_n_threads}.

What it means

Marlin tiles the output (N) dimension across thread groups; the per-partition output size (sum of output_partition_sizes, i.e. local N after TP sharding) must be divisible by the config's min_n_threads. If not, this ValueError fires at create_weights before tensors are allocated.

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:762

        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        output_size: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,
    ):
        del output_size  # Unused.
        weight_loader = extra_weight_attrs["weight_loader"]

        if params_dtype != torch.float16:
            raise ValueError(
                f"The params dtype must be float16, but got {params_dtype}"
            )

        # Validate output_size_per_partition
        output_size_per_partition = sum(output_partition_sizes)
        if output_size_per_partition % self.quant_config.min_n_threads != 0:
            raise ValueError(
                f"Weight output_size_per_partition = "
                f"{output_size_per_partition} is not divisible by "
                f"min_n_threads = {self.quant_config.min_n_threads}."
            )
        if output_size_per_partition % self.quant_config.pack_factor != 0:
            raise ValueError(
                f"Weight output_size_per_partition = "
                f"{output_size_per_partition} is not divisible by "
                f"pack_factor = {self.quant_config.pack_factor}."
            )

        # Validate input_size_per_partition
        if input_size_per_partition % self.quant_config.min_k_threads != 0:
            raise ValueError(
                f"Weight input_size_per_partition = "
                f"{input_size_per_partition} is not divisible by "
                f"min_k_threads = {self.quant_config.min_k_threads}."
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Reduce tensor_parallel_size to the largest power-of-two divisor of the output dim (1, 2, 4, 8)
2. Fall back to --quantization gptq (non-Marlin) which has weaker shape constraints
3. Verify intermediate_size / TP is divisible by min_n_threads (typically 64)

Example fix

# before
--tensor-parallel-size 6   # 11008/6 not divisible by min_n_threads
# after
--tensor-parallel-size 2

min_n = 64  # GPTQ Marlin min_n_threads
assert (intermediate_size // tp) % min_n == 0, "local N breaks Marlin thread tiling"

When it happens

Trigger: A tensor_parallel_size that leaves output_size_per_partition not divisible by min_n_threads (commonly 64 for Marlin); MoE experts sharded so local expert N breaks the divisibility.

Common situations: Non-power-of-two TP degrees (3, 6, 7); small models whose intermediate size barely exceeds min_n_threads.

Related errors

• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_size_per_partition = {output_size_per_partitio
• Weight input_size_per_partition = {input_size_per_partition}
• num_bits must be 4 or 8, got {}
• Currently, only group size 128 and -1 (channelwise) is suppo

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/798ace11d70a48d7. Report an issue: GitHub.