What causes "Inkling only supports group size 16 for NVFP4" in sgl-project/sglang?

Re-quantize the checkpoint with group_size=16 in the quantization_config

sgl-project/sglang · error · ValueError

Inkling only supports group size 16 for NVFP4

Error message

Inkling only supports group size 16 for NVFP4

What it means

The Inkling NVFP4 quantization config only supports group size 16 (the NVFP4 block format). __init__ calls the parent constructor then raises if group_size != 16, because dim1/dim2 group math assumes 16-wide FP4 groups.

Solutions

Re-quantize the checkpoint with group_size=16 in the quantization_config
If you control the config, set quantization_config['group_size'] = 16 before loading

Example fix

# before
quantization_config = {"quant_method": "nvfp4", "group_size": 64}
# after
quantization_config = {"quant_method": "nvfp4", "group_size": 16}

Defensive patterns

Strategy: validation

Validate before calling

assert quantization_config['group_size'] == 16, 'Inkling NVFP4 requires group_size 16'

Prevention

Pin group_size=16 in your quantization recipes for Inkling models
Validate the HF quantization_config before launching the server

When it happens

Trigger: Constructing the Inkling NVFP4 config with a group_size other than 16, typically from a checkpoint's HF quantization_config (e.g. group_size 32 or 64).

Common situations: Quantizing/re-exporting the model with a non-standard group size; mixing a generic NVFP4 recipe with Inkling's required settings.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0d613400b21061b3. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/models/inkling_common/quantization/config.py:146

        kv_cache_quant_algo: str | None = None,
        group_size: int | None = None,
        exclude_modules: list[str] | None = None,
        packed_modules_mapping: dict[str, list[str]] | None = None,
        # New Inkling args
        scales_2d: bool = False,
        moe_ep_size: int = 1,
        nvfp4_moe_backend: str = "trtllm-routed",
    ) -> None:
        # unfortunately parent types are completely incorrect
        super().__init__(
            is_checkpoint_nvfp4_serialized=is_checkpoint_nvfp4_serialized,
            kv_cache_quant_algo=kv_cache_quant_algo,  # type: ignore[reportArgumentType]
            group_size=group_size,  # type: ignore[reportArgumentType]
            exclude_modules=exclude_modules,  # type: ignore[reportArgumentType]
            packed_modules_mapping=packed_modules_mapping,
        )
        if group_size != 16:
            raise ValueError("Inkling only supports group size 16 for NVFP4")
        if scales_2d:
            self.dim1_group_size = group_size
        else:
            self.dim1_group_size = 1
        self.dim2_group_size = group_size
        self.moe_ep_size = moe_ep_size
        self.nvfp4_moe_backend = nvfp4_moe_backend

    @classmethod
    def get_name(cls) -> str:
        return "inkling_nvfp4"

    def get_quant_method(
        self, layer: torch.nn.Module, prefix: str
    ) -> QuantizeMethodBase | None:
        """Map layers to Inkling-compatible quant methods."""

        # hidden to avoid circular imports

View on GitHub (pinned to 0132848349)