sgl-project/sglang · error · ValueError

Inkling only supports group size 16 for NVFP4

Error message

Inkling only supports group size 16 for NVFP4

What it means

The Inkling NVFP4 quantization config only supports group size 16 (the NVFP4 block format). __init__ calls the parent constructor then raises if group_size != 16, because dim1/dim2 group math assumes 16-wide FP4 groups.

Source

Thrown at python/sglang/srt/models/inkling_common/quantization/config.py:146

        kv_cache_quant_algo: str | None = None,
        group_size: int | None = None,
        exclude_modules: list[str] | None = None,
        packed_modules_mapping: dict[str, list[str]] | None = None,
        # New Inkling args
        scales_2d: bool = False,
        moe_ep_size: int = 1,
        nvfp4_moe_backend: str = "trtllm-routed",
    ) -> None:
        # unfortunately parent types are completely incorrect
        super().__init__(
            is_checkpoint_nvfp4_serialized=is_checkpoint_nvfp4_serialized,
            kv_cache_quant_algo=kv_cache_quant_algo,  # type: ignore[reportArgumentType]
            group_size=group_size,  # type: ignore[reportArgumentType]
            exclude_modules=exclude_modules,  # type: ignore[reportArgumentType]
            packed_modules_mapping=packed_modules_mapping,
        )
        if group_size != 16:
            raise ValueError("Inkling only supports group size 16 for NVFP4")
        if scales_2d:
            self.dim1_group_size = group_size
        else:
            self.dim1_group_size = 1
        self.dim2_group_size = group_size
        self.moe_ep_size = moe_ep_size
        self.nvfp4_moe_backend = nvfp4_moe_backend

    @classmethod
    def get_name(cls) -> str:
        return "inkling_nvfp4"

    def get_quant_method(
        self, layer: torch.nn.Module, prefix: str
    ) -> QuantizeMethodBase | None:
        """Map layers to Inkling-compatible quant methods."""

        # hidden to avoid circular imports

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize the checkpoint with group_size=16 in the quantization_config
2. If you control the config, set quantization_config['group_size'] = 16 before loading

Example fix

# before
quantization_config = {"quant_method": "nvfp4", "group_size": 64}
# after
quantization_config = {"quant_method": "nvfp4", "group_size": 16}

assert quantization_config['group_size'] == 16, 'Inkling NVFP4 requires group_size 16'

When it happens

Trigger: Constructing the Inkling NVFP4 config with a group_size other than 16, typically from a checkpoint's HF quantization_config (e.g. group_size 32 or 64).

Common situations: Quantizing/re-exporting the model with a non-standard group size; mixing a generic NVFP4 recipe with Inkling's required settings.

Related errors

• InklingNvfp4MoEMethod is the dense shared-expert method; rou
• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.
• Type must match: {self.a_dtype} != {self.b_dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0d613400b21061b3. Report an issue: GitHub.