sgl-project/sglang · error · ValueError

Marlin kernels require group quantization or channelwise qua

Error message

Marlin kernels require group quantization or channelwise quantization, but found no group size and strategy is not channelwise.

What it means

Marlin wNa16 kernels need either group-wise quantization (a group_size) or channelwise strategy. A checkpoint with no group_size and a non-channel strategy (e.g. per-tensor) cannot be repacked for Marlin, so scheme init fails immediately.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py:80

class CompressedTensorsWNA16(CompressedTensorsLinearScheme):
    _kernel_backends_being_used: set[str] = set()

    def __init__(self,
                 strategy: str,
                 num_bits: int,
                 group_size: Optional[int] = None,
                 symmetric: Optional[bool] = True,
                 actorder: Optional[ActivationOrdering] = None):

        self.pack_factor = 32 // num_bits
        self.strategy = strategy
        self.symmetric = symmetric
        self.group_size = -1 if group_size is None else group_size
        self.has_g_idx = actorder == ActivationOrdering.GROUP

        if self.group_size == -1 and self.strategy != "channel":
            raise ValueError("Marlin kernels require group quantization or "
                             "channelwise quantization, but found no group "
                             "size and strategy is not channelwise.")

        if num_bits not in WNA16_SUPPORTED_TYPES_MAP:
            raise ValueError(
                f"Unsupported num_bits = {num_bits}. "
                f"Supported num_bits = {WNA16_SUPPORTED_TYPES_MAP.keys()}")

        self.quant_type = (WNA16_ZP_SUPPORTED_TYPES_MAP[num_bits]
                           if not self.symmetric else
                           WNA16_SUPPORTED_TYPES_MAP[num_bits])

    @classmethod
    def get_min_capability(cls) -> int:
        # ampere and up
        return 80

    def create_weights(self, layer: torch.nn.Module, output_size: int,

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize with group_size set (e.g. 32/64/128) or channelwise strategy
2. Verify config weights: {"strategy": "group", "group_size": 128}
3. If per-tensor is required, use a non-Marlin-compatible scheme or convert the checkpoint

Example fix

// before
"weights": {"num_bits": 4, "strategy": "tensor"}
// after
"weights": {"num_bits": 4, "strategy": "group", "group_size": 128}

w = cfg["quantization_config"]["weights"]
assert w.get("group_size") is not None or w["strategy"] == "channel"

def marlin_compatible(w):
    return w.get("group_size") is not None or w["strategy"] == "channel"

When it happens

Trigger: Loading a GPTQ/AWQ-style compressed-tensors model whose weights section has strategy 'tensor' and no group_size, routed to CompressedTensorsWNA16.

Common situations: Per-tensor INT4/INT8-quantized checkpoints mistakenly run through the Marlin path; quant configs missing the group_size field.

Related errors

• Pack: Only supports tensors with dimensions not greater than
• The Triton WNA16 MoE backend only supports symmetric INT4 gr
• Unsupported weight strategy={self.strategy}, supported strat
• Unknown quantization strategy {self.strategy}
• Unsupported weight quantization strategy: {self.weight_quant

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9d41160134864526. Report an issue: GitHub.