sgl-project/sglang · error · ValueError

Weight input_size_per_partition = {input_size_per_partition}

Error message

Weight input_size_per_partition = {input_size_per_partition} is not divisible by min_thread_k = {GPTQ_MARLIN_MIN_THREAD_K}. Consider reducing tensor_parallel_size or running with --quantization gptq.

What it means

The K-side counterpart in verify_marlin_supports_shape: input_size_per_partition must be divisible by GPTQ_MARLIN_MIN_THREAD_K (typically 256), the kernel's minimum K tile. TP sharding that leaves local K below or misaligned with this tile raises the error, with the same suggested fixes (reduce TP or fall back to --quantization gptq).

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:197

    output_size_per_partition: int,
    input_size_per_partition: int,
    input_size: int,
    group_size: int,
) -> None:

    # Validate output_size_per_partition
    if output_size_per_partition % GPTQ_MARLIN_MIN_THREAD_N != 0:
        raise ValueError(
            f"Weight output_size_per_partition = "
            f"{output_size_per_partition} is not divisible by "
            f" min_thread_n = {GPTQ_MARLIN_MIN_THREAD_N}. "
            "Consider reducing tensor_parallel_size or running "
            "with --quantization gptq."
        )

    # Validate input_size_per_partition
    if input_size_per_partition % GPTQ_MARLIN_MIN_THREAD_K != 0:
        raise ValueError(
            f"Weight input_size_per_partition = "
            f"{input_size_per_partition} is not divisible "
            f"by min_thread_k = {GPTQ_MARLIN_MIN_THREAD_K}. "
            "Consider reducing tensor_parallel_size or running "
            "with --quantization gptq."
        )

    if group_size < input_size and input_size_per_partition % group_size != 0:
        raise ValueError(
            f"Weight input_size_per_partition = {input_size_per_partition}"
            f" is not divisible by group_size = {group_size}. "
            "Consider reducing tensor_parallel_size or running "
            "with --quantization gptq."
        )


def check_marlin_supports_shape(
    output_size_per_partition: int,

View on GitHub (pinned to 0132848349)

Solutions

Reduce tensor_parallel_size until local K is a multiple of GPTQ_MARLIN_MIN_THREAD_K
Run with --quantization gptq to skip the Marlin kernel entirely
Pre-check input_size // TP % GPTQ_MARLIN_MIN_THREAD_K == 0 before launching

Example fix

# before
--tensor-parallel-size 16  # local K 128 < min_thread_k 256
# after
--tensor-parallel-size 8

Defensive patterns

Strategy: fallback

Validate before calling

from sglang.srt.layers.quantization.marlin_utils import check_marlin_supports_shape
check_marlin_supports_shape(output_size // tp, input_size // tp, group_size)

Try / catch

try:
    check_marlin_supports_shape(out_pp, in_pp, gs)
except ValueError as e:
    if "min_thread_k" in str(e):
        server_args.tp_size //= 2  # retry with smaller TP
    else:
        raise

Prevention

Pre-check hidden_size // TP against GPTQ_MARLIN_MIN_THREAD_K
Keep --quantization gptq as a documented fallback for odd TP layouts

When it happens

Trigger: hidden_size / TP not divisible by GPTQ_MARLIN_MIN_THREAD_K, e.g. hidden 2048 with TP=16 giving local K=128 < 256; QKV projection K partitions after sharding.

Common situations: Very high TP degrees on small models; quantized Llama-variants with small hidden sizes; automated TP sweep scripts.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/048b71b6ee1efafc. Report an issue: GitHub.