{"record":{"id":"048b71b6ee1efafc","repo":"sgl-project/sglang","slug":"weight-input-size-per-partition-input-size-per-048b71","errorCode":null,"errorMessage":"Weight input_size_per_partition = {input_size_per_partition} is not divisible by min_thread_k = {GPTQ_MARLIN_MIN_THREAD_K}. Consider reducing tensor_parallel_size or running with --quantization gptq.","messagePattern":"Weight input_size_per_partition = (.+?) is not divisible by min_thread_k = (.+?)\\. Consider reducing tensor_parallel_size or running with --quantization gptq\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/marlin_utils.py","lineNumber":197,"sourceCode":"    output_size_per_partition: int,\n    input_size_per_partition: int,\n    input_size: int,\n    group_size: int,\n) -> None:\n\n    # Validate output_size_per_partition\n    if output_size_per_partition % GPTQ_MARLIN_MIN_THREAD_N != 0:\n        raise ValueError(\n            f\"Weight output_size_per_partition = \"\n            f\"{output_size_per_partition} is not divisible by \"\n            f\" min_thread_n = {GPTQ_MARLIN_MIN_THREAD_N}. \"\n            \"Consider reducing tensor_parallel_size or running \"\n            \"with --quantization gptq.\"\n        )\n\n    # Validate input_size_per_partition\n    if input_size_per_partition % GPTQ_MARLIN_MIN_THREAD_K != 0:\n        raise ValueError(\n            f\"Weight input_size_per_partition = \"\n            f\"{input_size_per_partition} is not divisible \"\n            f\"by min_thread_k = {GPTQ_MARLIN_MIN_THREAD_K}. \"\n            \"Consider reducing tensor_parallel_size or running \"\n            \"with --quantization gptq.\"\n        )\n\n    if group_size < input_size and input_size_per_partition % group_size != 0:\n        raise ValueError(\n            f\"Weight input_size_per_partition = {input_size_per_partition}\"\n            f\" is not divisible by group_size = {group_size}. \"\n            \"Consider reducing tensor_parallel_size or running \"\n            \"with --quantization gptq.\"\n        )\n\n\ndef check_marlin_supports_shape(\n    output_size_per_partition: int,","sourceCodeStart":179,"sourceCodeEnd":215,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/marlin_utils.py#L179-L215","documentation":"The K-side counterpart in verify_marlin_supports_shape: input_size_per_partition must be divisible by GPTQ_MARLIN_MIN_THREAD_K (typically 256), the kernel's minimum K tile. TP sharding that leaves local K below or misaligned with this tile raises the error, with the same suggested fixes (reduce TP or fall back to --quantization gptq).","triggerScenarios":"hidden_size / TP not divisible by GPTQ_MARLIN_MIN_THREAD_K, e.g. hidden 2048 with TP=16 giving local K=128 < 256; QKV projection K partitions after sharding.","commonSituations":"Very high TP degrees on small models; quantized Llama-variants with small hidden sizes; automated TP sweep scripts.","solutions":["Reduce tensor_parallel_size until local K is a multiple of GPTQ_MARLIN_MIN_THREAD_K","Run with --quantization gptq to skip the Marlin kernel entirely","Pre-check input_size // TP % GPTQ_MARLIN_MIN_THREAD_K == 0 before launching"],"exampleFix":"# before\n--tensor-parallel-size 16  # local K 128 < min_thread_k 256\n# after\n--tensor-parallel-size 8","handlingStrategy":"fallback","validationCode":"from sglang.srt.layers.quantization.marlin_utils import check_marlin_supports_shape\ncheck_marlin_supports_shape(output_size // tp, input_size // tp, group_size)","typeGuard":null,"tryCatchPattern":"try:\n    check_marlin_supports_shape(out_pp, in_pp, gs)\nexcept ValueError as e:\n    if \"min_thread_k\" in str(e):\n        server_args.tp_size //= 2  # retry with smaller TP\n    else:\n        raise","preventionTips":["Pre-check hidden_size // TP against GPTQ_MARLIN_MIN_THREAD_K","Keep --quantization gptq as a documented fallback for odd TP layouts"],"tags":["marlin","gptq","tensor-parallel","shape-validation"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}