What causes "Each permutation group must reside on the same gpu" in sgl-project/sglang?

Reduce tensor_parallel_size to a power-of-two divisor so local N respects tile alignment

sgl-project/sglang · error · ValueError

Each permutation group must reside on the same gpu

Error message

Each permutation group must reside on the same gpu

What it means

Marlin's permutation-based weight layout maps tiles of tile_size^2 elements per int32 permutation unit; the shard's output_size_per_partition must be divisible by perm_len // tile_size^2 (num_tiles_per_perm) so no permutation group straddles two GPUs. create_weights raises this generic ValueError when TP sharding splits a permutation group.

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:796

                f"{input_size_per_partition} is not divisible by "
                f"min_k_threads = {self.quant_config.min_k_threads}."
            )
        if (
            self.quant_config.group_size != -1
            and input_size_per_partition % self.quant_config.group_size != 0
        ):
            raise ValueError(
                f"Weight input_size_per_partition = "
                f"{input_size_per_partition} is not divisible by "
                f"group_size = {self.quant_config.group_size}."
            )

        # Check that we have at least 4 tiles horizontally in the shard
        num_tiles_per_perm = self.quant_config.perm_len // (
            self.quant_config.tile_size**2
        )
        if output_size_per_partition % num_tiles_per_perm != 0:
            raise ValueError("Each permutation group must reside on the same gpu")

        # Quantized 4Bit weights packed into Int32.
        qweight = PackedvLLMParameter(
            data=torch.empty(
                input_size_per_partition // self.quant_config.tile_size,
                output_size_per_partition
                * self.quant_config.tile_size
                // self.quant_config.pack_factor,
                device="cuda",
                dtype=torch.int32,
            ),
            input_dim=0,
            output_dim=1,
            packed_dim=1,
            packed_factor=self.quant_config.pack_factor,
            marlin_tile_size=self.quant_config.tile_size,
            weight_loader=weight_loader,
        )

View on GitHub (pinned to 0132848349)

Solutions

Reduce tensor_parallel_size to a power-of-two divisor so local N respects tile alignment
Fall back to --quantization gptq (dense kernels, no permutation layout)
Compute num_tiles_per_perm = perm_len // tile_size**2 and assert local N divisibility pre-launch

Example fix

# before
--tensor-parallel-size 6
# after
--tensor-parallel-size 2

Defensive patterns

Strategy: validation

Validate before calling

num_tiles_per_perm = perm_len // (tile_size ** 2)  # e.g. 1024 // 256 = 4
assert output_size_per_partition % num_tiles_per_perm == 0, "permutation group straddles GPUs"

Prevention

Use power-of-two TP with Marlin permutation layouts
Compute tile alignment in a launch-time sanity check

When it happens

Trigger: output_size_per_partition % num_tiles_per_perm != 0 after TP sharding — typically with tile_size 16 and perm_len 1024 giving 4 tiles per perm, so local N must be a multiple of 4*16 alignment; occurs with odd TP degrees or small N.

Common situations: Serving quantized models on 3/5/6/7 GPU TP layouts; MoE experts with small per-expert N dims.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/854072fb3447ee64. Report an issue: GitHub.