sgl-project/sglang · error · ValueError

W4AFP8 shape_k = {shape_k} must be divisible by group_size =

Error message

W4AFP8 shape_k = {shape_k} must be divisible by group_size = {self.group_size}. Choose a tensor-parallel configuration whose local K dimension preserves quantization groups.

What it means

Humming W4AFP8 requires the layer's local K dimension to be divisible by group_size (and by 8) so quantization groups tile the sharded weight; get_tensors_attrs raises with guidance that the tensor-parallel configuration broke the groups.

Source

Thrown at python/sglang/srt/layers/quantization/humming.py:250

            or isinstance(group_size, bool)
            or group_size <= 0
        ):
            raise ValueError(
                f"W4AFP8 group_size must be a positive integer, got {group_size!r}."
            )
        self.group_size = group_size

    def get_tensors_attrs(
        self,
        shape_n: int,
        shape_k: int,
        param_dtype: torch.dtype,
        num_experts: int | None = None,
        has_bias: bool = False,
        stack_size: int = 1,
    ) -> dict[str, dict[str, Any]]:
        if shape_k % self.group_size != 0:
            raise ValueError(
                f"W4AFP8 shape_k = {shape_k} must be divisible by group_size = "
                f"{self.group_size}. Choose a tensor-parallel configuration whose "
                "local K dimension preserves quantization groups."
            )
        if shape_k % 8 != 0:
            raise ValueError(
                f"W4AFP8 shape_k = {shape_k} must be divisible by 8 for int32 "
                "packed-weight storage."
            )

        tensors_attrs = {
            "weight": {
                "shape": (shape_n, shape_k // 2),
                "dtype": torch.int8,
                "extra_attrs": {"output_dim": 0, "input_dim": 1},
            },
            "weight_scale_inv": {
                "shape": (shape_n, shape_k // self.group_size),

View on GitHub (pinned to 0132848349)

Solutions

1. Choose a tensor parallel size where (hidden_size / tp) % group_size == 0 — typically tp in {1,2,4,8} for group_size 128
2. Use tp=1 if the model has unusual K dimensions
3. Requantize with a smaller group_size (e.g. 32/64) that divides the local K

Example fix

# before
--tp 3   # 11008/3 = 3669.33 not divisible by 128

# after
--tp 2   # 5504 % 128 == 0

local_k = hidden_size // tp_size
assert local_k % group_size == 0 and local_k % 8 == 0, (
    f"local K {local_k} breaks groups of {group_size}; choose tp in {{1,2,4,8}}")

def humming_tp_ok(shape_k: int, group: int, tp: int) -> bool:
    local = shape_k // tp
    return shape_k % tp == 0 and local % group == 0 and local % 8 == 0

When it happens

Trigger: get_tensors_attrs / create_weights with shape_k % group_size != 0 — e.g. hidden_size 11008 with tp=3 and group_size 128, or MoE expert dims that don't shard into group multiples; a second check rejects shape_k not divisible by 8.

Common situations: Odd/non-power-of-two TP degrees on Humming-quantized models; also small router/projection layers whose K is naturally not a multiple of the group size.

Related errors

• Weight output_partition_size = {output_partition_size} is no
• Comfy W4A8 layer {prefix!r} has incompatible weight/scale sh
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• The input size is not aligned with the quantized weight shap

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/719d6574685fb36c. Report an issue: GitHub.