{"record":{"id":"2df0be80ebb262f9","repo":"sgl-project/sglang","slug":"w4afp8-shape-k-shape-k-must-be-divisible-by-8","errorCode":null,"errorMessage":"W4AFP8 shape_k = {shape_k} must be divisible by 8 for int32 packed-weight storage.","messagePattern":"W4AFP8 shape_k = (.+?) must be divisible by 8 for int32 packed-weight storage\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/humming.py","lineNumber":256,"sourceCode":"        self.group_size = group_size\n\n    def get_tensors_attrs(\n        self,\n        shape_n: int,\n        shape_k: int,\n        param_dtype: torch.dtype,\n        num_experts: int | None = None,\n        has_bias: bool = False,\n        stack_size: int = 1,\n    ) -> dict[str, dict[str, Any]]:\n        if shape_k % self.group_size != 0:\n            raise ValueError(\n                f\"W4AFP8 shape_k = {shape_k} must be divisible by group_size = \"\n                f\"{self.group_size}. Choose a tensor-parallel configuration whose \"\n                \"local K dimension preserves quantization groups.\"\n            )\n        if shape_k % 8 != 0:\n            raise ValueError(\n                f\"W4AFP8 shape_k = {shape_k} must be divisible by 8 for int32 \"\n                \"packed-weight storage.\"\n            )\n\n        tensors_attrs = {\n            \"weight\": {\n                \"shape\": (shape_n, shape_k // 2),\n                \"dtype\": torch.int8,\n                \"extra_attrs\": {\"output_dim\": 0, \"input_dim\": 1},\n            },\n            \"weight_scale_inv\": {\n                \"shape\": (shape_n, shape_k // self.group_size),\n                \"dtype\": param_dtype,\n                \"extra_attrs\": {\n                    \"output_dim\": 0,\n                    \"input_dim\": 1,\n                    \"scale_type\": \"group\",\n                },","sourceCodeStart":238,"sourceCodeEnd":274,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/humming.py#L238-L274","documentation":"W4AFP8 quantization packs four 4-bit weights into each int32 storage word, so the local (per-rank) K dimension of the linear layer must be a multiple of 8. get_tensors_attrs raises this at weight-creation time when shape_k % 8 != 0, which almost always means the tensor-parallel (TP) sharding split the global K dimension into a rank-local size that breaks the packing.","triggerScenarios":"Loading a W4AFP8 (Humming) quantized model with a tensor_parallel_size that does not divide the layer's input/K dimension into a multiple of 8; e.g. a hidden size of 4096 with an odd TP degree producing shape_k like 512.5->non-integer groups, or interleaved/MoE shards whose local K is not divisible by 8.","commonSituations":"Running --tensor-parallel-size 3 or 6 on a model whose intermediate/hidden dims are power-of-two multiples; mixing W4AFP8 checkpoints with TP configs validated only for FP16/BF16; custom models with unusual K dims.","solutions":["Pick a tensor_parallel_size that divides the layer K dimension so the local shape_k is divisible by 8 (powers of two: 1, 2, 4, 8 are safest)","Run with TP=1 to confirm the checkpoint loads, then increase TP one step at a time checking shape_k divisibility","If the model architecture allows, use a different quantization format without the 8-way packing constraint"],"exampleFix":"# before\npython -m sglang.launch_server --model w4afp8-model --tensor-parallel-size 3\n# ValueError: shape_k = 1365 must be divisible by 8 ...\n\n# after\npython -m sglang.launch_server --model w4afp8-model --tensor-parallel-size 4","handlingStrategy":"validation","validationCode":"tp = server_args.tp_size\nhidden = model_config.hidden_size\nassert (hidden // tp) % 8 == 0, f\"local K {hidden//tp} not divisible by 8 for W4AFP8\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Only use power-of-two TP degrees with W4AFP8 checkpoints","Add a startup assertion on hidden_size // tp % 8 in your launch wrapper"],"tags":["quantization","w4afp8","humming","tensor-parallel","shape-validation"],"backgroundTag":"quantization-shape-divisibility","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}