{"record":{"id":"854072fb3447ee64","repo":"sgl-project/sglang","slug":"each-permutation-group-must-reside-on-the-same-gpu","errorCode":null,"errorMessage":"Each permutation group must reside on the same gpu","messagePattern":"Each permutation group must reside on the same gpu","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/marlin_utils.py","lineNumber":796,"sourceCode":"                f\"{input_size_per_partition} is not divisible by \"\n                f\"min_k_threads = {self.quant_config.min_k_threads}.\"\n            )\n        if (\n            self.quant_config.group_size != -1\n            and input_size_per_partition % self.quant_config.group_size != 0\n        ):\n            raise ValueError(\n                f\"Weight input_size_per_partition = \"\n                f\"{input_size_per_partition} is not divisible by \"\n                f\"group_size = {self.quant_config.group_size}.\"\n            )\n\n        # Check that we have at least 4 tiles horizontally in the shard\n        num_tiles_per_perm = self.quant_config.perm_len // (\n            self.quant_config.tile_size**2\n        )\n        if output_size_per_partition % num_tiles_per_perm != 0:\n            raise ValueError(\"Each permutation group must reside on the same gpu\")\n\n        # Quantized 4Bit weights packed into Int32.\n        qweight = PackedvLLMParameter(\n            data=torch.empty(\n                input_size_per_partition // self.quant_config.tile_size,\n                output_size_per_partition\n                * self.quant_config.tile_size\n                // self.quant_config.pack_factor,\n                device=\"cuda\",\n                dtype=torch.int32,\n            ),\n            input_dim=0,\n            output_dim=1,\n            packed_dim=1,\n            packed_factor=self.quant_config.pack_factor,\n            marlin_tile_size=self.quant_config.tile_size,\n            weight_loader=weight_loader,\n        )","sourceCodeStart":778,"sourceCodeEnd":814,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/marlin_utils.py#L778-L814","documentation":"Marlin's permutation-based weight layout maps tiles of tile_size^2 elements per int32 permutation unit; the shard's output_size_per_partition must be divisible by perm_len // tile_size^2 (num_tiles_per_perm) so no permutation group straddles two GPUs. create_weights raises this generic ValueError when TP sharding splits a permutation group.","triggerScenarios":"output_size_per_partition % num_tiles_per_perm != 0 after TP sharding — typically with tile_size 16 and perm_len 1024 giving 4 tiles per perm, so local N must be a multiple of 4*16 alignment; occurs with odd TP degrees or small N.","commonSituations":"Serving quantized models on 3/5/6/7 GPU TP layouts; MoE experts with small per-expert N dims.","solutions":["Reduce tensor_parallel_size to a power-of-two divisor so local N respects tile alignment","Fall back to --quantization gptq (dense kernels, no permutation layout)","Compute num_tiles_per_perm = perm_len // tile_size**2 and assert local N divisibility pre-launch"],"exampleFix":"# before\n--tensor-parallel-size 6\n# after\n--tensor-parallel-size 2","handlingStrategy":"validation","validationCode":"num_tiles_per_perm = perm_len // (tile_size ** 2)  # e.g. 1024 // 256 = 4\nassert output_size_per_partition % num_tiles_per_perm == 0, \"permutation group straddles GPUs\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use power-of-two TP with Marlin permutation layouts","Compute tile alignment in a launch-time sanity check"],"tags":["marlin","tensor-parallel","tile-alignment","shape-validation"],"backgroundTag":"tensor-parallel-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}