sgl-project/sglang · error · NotImplementedError

GPTQ act_order on XPU requires each group_size block of inpu

Error message

GPTQ act_order on XPU requires each group_size block of input channels to map to a single group, but this shard splits a group across the K boundary.{tp_hint}

What it means

XPU GPTQ kernels with act_order (gidx) require each group_size block of input channels to stay within one quantization group after tensor-parallel sharding. When TP sharding splits the K dimension such that a group straddles the shard boundary, per-group reordering cannot be applied correctly, so weight post-processing raises NotImplementedError. The error message includes the detected tp_size and suggests running unsharded.

Source

Thrown at python/sglang/srt/hardware_backend/xpu/quantization/gptq_kernels.py:71

            g_idx = layer.g_idx.data
            if g_idx.numel() != k:
                raise ValueError(
                    "GPTQ act_order on XPU expects a per-channel g_idx of length "
                    f"K={k}, got {g_idx.numel()}."
                )
            # Sort K by group id so groups become contiguous gs-blocks.
            act_perm = torch.argsort(g_idx, stable=True).to(torch.int64)
            codes = codes[act_perm, :]
            sorted_g = g_idx[act_perm].to(torch.int64)
            blocks = sorted_g.view(-1, group_size)
            if not torch.equal(blocks, blocks[:, :1].expand_as(blocks)):
                tp_size = get_parallel().tp_size
                tp_hint = (
                    f" Got tp_size={tp_size}; please use --tp-size 1."
                    if tp_size > 1
                    else ""
                )
                raise NotImplementedError(
                    "GPTQ act_order on XPU requires each group_size block of "
                    "input channels to map to a single group, but this shard "
                    "splits a group across the K boundary." + tp_hint
                )
            # Reorder scales/zeros to follow the block group order.
            block_gid = blocks[:, 0]  # [num_blocks]
            scales = scales[block_gid]
            zp = zp[block_gid]

        codes = codes.t().contiguous()  # [N, K]
        qweight_uint8 = pack_int4_to_uint8(codes)  # [N, K // 2]
        qweight_packed = torch.ops.aten._convert_weight_to_int4pack(
            qweight_uint8, 8
        )  # [N, K // 8] int32

        replace_parameter(layer, "qweight", qweight_packed)
        layer.register_parameter(
            "xpu_scales",

View on GitHub (pinned to 0132848349)

Solutions

1. Use --tp-size 1 (as the tp_hint suggests) so the full K dimension lives in one shard
2. Pick a tp_size such that (num_input_channels / tp_size) is an integer multiple of the checkpoint's group_size
3. Use a GPTQ checkpoint without act_order/desc_act, or a quantization format supported on XPU with TP (e.g. unsharded-compatible formats)
4. Report/request upstream support for boundary-splitting groups in the XPU act_order path

Example fix

# before
python -m sglang.launch_server --model gptq-model --tp-size 4  # raises NotImplementedError

# after
python -m sglang.launch_server --model gptq-model --tp-size 1

group_size = ckpt_quant_cfg.group_size  # e.g. 128
k = num_input_channels
tp = get_parallel().tp_size
if act_order and tp > 1 and (k // tp) % group_size != 0:
    raise SystemExit(
        f"GPTQ act_order on XPU with tp={tp} splits a group "
        f"(K/tp={k//tp}, group_size={group_size}); use --tp-size 1"
    )

When it happens

Trigger: Loading a GPTQ checkpoint with act_order=True (desc_act) on Intel XPU with --tp-size > 1 where (input_channels / tp_size) is not aligned to the checkpoint's group_size (e.g. group_size=128 and the per-shard K is not a multiple of 128). Raised from process_weights_after_loading in gptq_kernels.py:71 during model weight loading.

Common situations: Running a GPTQ act-order model (e.g. TheBloke GPTQ with desc_act) on Intel GPUs with tensor parallelism where K is not divisible by tp_size*group_size; switching a working single-GPU config to multi-GPU TP; newer checkpoints with unusual group_size values.

Related errors

• bitsandbytes 4-bit TP only supports column-parallel output s
• bitsandbytes 4-bit TP does not support nested quant states.
• The input size is not aligned with the quantized weight shap
• QVGPackedCausalKVCache does not support pinned-sink (longliv
• bitsandbytes 4-bit TP shard is not aligned to quantization b

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4c1ad0cbdf18bbc2. Report an issue: GitHub.