sgl-project/sglang · error · ValueError

W4AFP8 shape_k = {shape_k} must be divisible by 8 for int32

Error message

W4AFP8 shape_k = {shape_k} must be divisible by 8 for int32 packed-weight storage.

What it means

W4AFP8 quantization packs four 4-bit weights into each int32 storage word, so the local (per-rank) K dimension of the linear layer must be a multiple of 8. get_tensors_attrs raises this at weight-creation time when shape_k % 8 != 0, which almost always means the tensor-parallel (TP) sharding split the global K dimension into a rank-local size that breaks the packing.

Source

Thrown at python/sglang/srt/layers/quantization/humming.py:256

        self.group_size = group_size

    def get_tensors_attrs(
        self,
        shape_n: int,
        shape_k: int,
        param_dtype: torch.dtype,
        num_experts: int | None = None,
        has_bias: bool = False,
        stack_size: int = 1,
    ) -> dict[str, dict[str, Any]]:
        if shape_k % self.group_size != 0:
            raise ValueError(
                f"W4AFP8 shape_k = {shape_k} must be divisible by group_size = "
                f"{self.group_size}. Choose a tensor-parallel configuration whose "
                "local K dimension preserves quantization groups."
            )
        if shape_k % 8 != 0:
            raise ValueError(
                f"W4AFP8 shape_k = {shape_k} must be divisible by 8 for int32 "
                "packed-weight storage."
            )

        tensors_attrs = {
            "weight": {
                "shape": (shape_n, shape_k // 2),
                "dtype": torch.int8,
                "extra_attrs": {"output_dim": 0, "input_dim": 1},
            },
            "weight_scale_inv": {
                "shape": (shape_n, shape_k // self.group_size),
                "dtype": param_dtype,
                "extra_attrs": {
                    "output_dim": 0,
                    "input_dim": 1,
                    "scale_type": "group",
                },

View on GitHub (pinned to 0132848349)

Solutions

1. Pick a tensor_parallel_size that divides the layer K dimension so the local shape_k is divisible by 8 (powers of two: 1, 2, 4, 8 are safest)
2. Run with TP=1 to confirm the checkpoint loads, then increase TP one step at a time checking shape_k divisibility
3. If the model architecture allows, use a different quantization format without the 8-way packing constraint

Example fix

# before
python -m sglang.launch_server --model w4afp8-model --tensor-parallel-size 3
# ValueError: shape_k = 1365 must be divisible by 8 ...

# after
python -m sglang.launch_server --model w4afp8-model --tensor-parallel-size 4

tp = server_args.tp_size
hidden = model_config.hidden_size
assert (hidden // tp) % 8 == 0, f"local K {hidden//tp} not divisible by 8 for W4AFP8"

When it happens

Trigger: Loading a W4AFP8 (Humming) quantized model with a tensor_parallel_size that does not divide the layer's input/K dimension into a multiple of 8; e.g. a hidden size of 4096 with an odd TP degree producing shape_k like 512.5->non-integer groups, or interleaved/MoE shards whose local K is not divisible by 8.

Common situations: Running --tensor-parallel-size 3 or 6 on a model whose intermediate/hidden dims are power-of-two multiples; mixing W4AFP8 checkpoints with TP configs validated only for FP16/BF16; custom models with unusual K dims.

Related errors

• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_partition_size = {output_partition_size} is no
• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• W4AFP8 shape_k = {shape_k} must be divisible by group_size =

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/2df0be80ebb262f9. Report an issue: GitHub.