{"record":{"id":"719d6574685fb36c","repo":"sgl-project/sglang","slug":"w4afp8-shape-k-shape-k-must-be-divisible-by-gr","errorCode":null,"errorMessage":"W4AFP8 shape_k = {shape_k} must be divisible by group_size = {self.group_size}. Choose a tensor-parallel configuration whose local K dimension preserves quantization groups.","messagePattern":"W4AFP8 shape_k = (.+?) must be divisible by group_size = (.+?)\\. Choose a tensor-parallel configuration whose local K dimension preserves quantization groups\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/humming.py","lineNumber":250,"sourceCode":"            or isinstance(group_size, bool)\n            or group_size <= 0\n        ):\n            raise ValueError(\n                f\"W4AFP8 group_size must be a positive integer, got {group_size!r}.\"\n            )\n        self.group_size = group_size\n\n    def get_tensors_attrs(\n        self,\n        shape_n: int,\n        shape_k: int,\n        param_dtype: torch.dtype,\n        num_experts: int | None = None,\n        has_bias: bool = False,\n        stack_size: int = 1,\n    ) -> dict[str, dict[str, Any]]:\n        if shape_k % self.group_size != 0:\n            raise ValueError(\n                f\"W4AFP8 shape_k = {shape_k} must be divisible by group_size = \"\n                f\"{self.group_size}. Choose a tensor-parallel configuration whose \"\n                \"local K dimension preserves quantization groups.\"\n            )\n        if shape_k % 8 != 0:\n            raise ValueError(\n                f\"W4AFP8 shape_k = {shape_k} must be divisible by 8 for int32 \"\n                \"packed-weight storage.\"\n            )\n\n        tensors_attrs = {\n            \"weight\": {\n                \"shape\": (shape_n, shape_k // 2),\n                \"dtype\": torch.int8,\n                \"extra_attrs\": {\"output_dim\": 0, \"input_dim\": 1},\n            },\n            \"weight_scale_inv\": {\n                \"shape\": (shape_n, shape_k // self.group_size),","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/humming.py#L232-L268","documentation":"Humming W4AFP8 requires the layer's local K dimension to be divisible by group_size (and by 8) so quantization groups tile the sharded weight; get_tensors_attrs raises with guidance that the tensor-parallel configuration broke the groups.","triggerScenarios":"get_tensors_attrs / create_weights with shape_k % group_size != 0 — e.g. hidden_size 11008 with tp=3 and group_size 128, or MoE expert dims that don't shard into group multiples; a second check rejects shape_k not divisible by 8.","commonSituations":"Odd/non-power-of-two TP degrees on Humming-quantized models; also small router/projection layers whose K is naturally not a multiple of the group size.","solutions":["Choose a tensor parallel size where (hidden_size / tp) % group_size == 0 — typically tp in {1,2,4,8} for group_size 128","Use tp=1 if the model has unusual K dimensions","Requantize with a smaller group_size (e.g. 32/64) that divides the local K"],"exampleFix":"# before\n--tp 3   # 11008/3 = 3669.33 not divisible by 128\n\n# after\n--tp 2   # 5504 % 128 == 0","handlingStrategy":"validation","validationCode":"local_k = hidden_size // tp_size\nassert local_k % group_size == 0 and local_k % 8 == 0, (\n    f\"local K {local_k} breaks groups of {group_size}; choose tp in {{1,2,4,8}}\")","typeGuard":"def humming_tp_ok(shape_k: int, group: int, tp: int) -> bool:\n    local = shape_k // tp\n    return shape_k % tp == 0 and local % group == 0 and local % 8 == 0","tryCatchPattern":null,"preventionTips":["For group_size 128, only use TP degrees that divide hidden_size into 128-multiples","Add a deployment-time divisibility check (K % group == 0 and K % 8 == 0) per layer"],"tags":["humming","tensor-parallel","shape-mismatch","group-size","quantization"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}