sgl-project/sglang · error · ValueError

Weight output_size_per_partition = {output_size_per_partitio

Error message

Weight output_size_per_partition = {output_size_per_partition} is not divisible by  min_thread_n = {GPTQ_MARLIN_MIN_THREAD_N}. Consider reducing tensor_parallel_size or running with --quantization gptq.

What it means

verify_marlin_supports_shape checks that the TP-local output dimension is divisible by GPTQ_MARLIN_MIN_THREAD_N (the kernel's minimum N tiling, commonly 64). The error message itself suggests the two remedies: reduce tensor_parallel_size, or run with --quantization gptq to use the non-Marlin GEMM path. Called from create_weights and the pre-flight check_marlin_supports_shape.

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:187

def verify_marlin_supported(
    quant_type: ScalarType, group_size: int, has_zp: bool = False
) -> None:
    cond, err_msg = _check_marlin_supported(quant_type, group_size, has_zp)
    if not cond:
        assert err_msg is not None
        raise ValueError(err_msg)


def verify_marlin_supports_shape(
    output_size_per_partition: int,
    input_size_per_partition: int,
    input_size: int,
    group_size: int,
) -> None:

    # Validate output_size_per_partition
    if output_size_per_partition % GPTQ_MARLIN_MIN_THREAD_N != 0:
        raise ValueError(
            f"Weight output_size_per_partition = "
            f"{output_size_per_partition} is not divisible by "
            f" min_thread_n = {GPTQ_MARLIN_MIN_THREAD_N}. "
            "Consider reducing tensor_parallel_size or running "
            "with --quantization gptq."
        )

    # Validate input_size_per_partition
    if input_size_per_partition % GPTQ_MARLIN_MIN_THREAD_K != 0:
        raise ValueError(
            f"Weight input_size_per_partition = "
            f"{input_size_per_partition} is not divisible "
            f"by min_thread_k = {GPTQ_MARLIN_MIN_THREAD_K}. "
            "Consider reducing tensor_parallel_size or running "
            "with --quantization gptq."
        )

    if group_size < input_size and input_size_per_partition % group_size != 0:

View on GitHub (pinned to 0132848349)

Solutions

Reduce tensor_parallel_size (e.g. to 1, 2, 4) so local N is divisible by GPTQ_MARLIN_MIN_THREAD_N
Run with --quantization gptq to bypass Marlin
Use check_marlin_supports_shape in a pre-launch probe to select TP automatically

Example fix

# before
--tensor-parallel-size 8   # local N 512 not divisible by 64*... per kernel constraint
# after
--tensor-parallel-size 4 --quantization marlin   # or: --quantization gptq

Defensive patterns

Strategy: fallback

Validate before calling

from sglang.srt.layers.quantization.marlin_utils import check_marlin_supports_shape
check_marlin_supports_shape(output_size // tp, input_size // tp, group_size)  # raises early with guidance

Try / catch

try:
    check_marlin_supports_shape(out_pp, in_pp, gs)
except ValueError:
    server_args.quantization = "gptq"  # fall back to non-Marlin path

Prevention

Run check_marlin_supports_shape before launching Marlin models
Auto-select the largest power-of-two TP that satisfies thread-N/K divisibility

When it happens

Trigger: GPTQ Marlin serving with a TP degree making output_size_per_partition < or not divisible by GPTQ_MARLIN_MIN_THREAD_N; small models or heavily sharded MoE experts.

Common situations: Testing quantized small models on many GPUs; TP=6/7 style layouts; CI matrix runs sweeping TP sizes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/173dddc198fa5cc9. Report an issue: GitHub.