{"record":{"id":"0d613400b21061b3","repo":"sgl-project/sglang","slug":"inkling-only-supports-group-size-16-for-nvfp4","errorCode":null,"errorMessage":"Inkling only supports group size 16 for NVFP4","messagePattern":"Inkling only supports group size 16 for NVFP4","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/inkling_common/quantization/config.py","lineNumber":146,"sourceCode":"        kv_cache_quant_algo: str | None = None,\n        group_size: int | None = None,\n        exclude_modules: list[str] | None = None,\n        packed_modules_mapping: dict[str, list[str]] | None = None,\n        # New Inkling args\n        scales_2d: bool = False,\n        moe_ep_size: int = 1,\n        nvfp4_moe_backend: str = \"trtllm-routed\",\n    ) -> None:\n        # unfortunately parent types are completely incorrect\n        super().__init__(\n            is_checkpoint_nvfp4_serialized=is_checkpoint_nvfp4_serialized,\n            kv_cache_quant_algo=kv_cache_quant_algo,  # type: ignore[reportArgumentType]\n            group_size=group_size,  # type: ignore[reportArgumentType]\n            exclude_modules=exclude_modules,  # type: ignore[reportArgumentType]\n            packed_modules_mapping=packed_modules_mapping,\n        )\n        if group_size != 16:\n            raise ValueError(\"Inkling only supports group size 16 for NVFP4\")\n        if scales_2d:\n            self.dim1_group_size = group_size\n        else:\n            self.dim1_group_size = 1\n        self.dim2_group_size = group_size\n        self.moe_ep_size = moe_ep_size\n        self.nvfp4_moe_backend = nvfp4_moe_backend\n\n    @classmethod\n    def get_name(cls) -> str:\n        return \"inkling_nvfp4\"\n\n    def get_quant_method(\n        self, layer: torch.nn.Module, prefix: str\n    ) -> QuantizeMethodBase | None:\n        \"\"\"Map layers to Inkling-compatible quant methods.\"\"\"\n\n        # hidden to avoid circular imports","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/inkling_common/quantization/config.py#L128-L164","documentation":"The Inkling NVFP4 quantization config only supports group size 16 (the NVFP4 block format). __init__ calls the parent constructor then raises if group_size != 16, because dim1/dim2 group math assumes 16-wide FP4 groups.","triggerScenarios":"Constructing the Inkling NVFP4 config with a group_size other than 16, typically from a checkpoint's HF quantization_config (e.g. group_size 32 or 64).","commonSituations":"Quantizing/re-exporting the model with a non-standard group size; mixing a generic NVFP4 recipe with Inkling's required settings.","solutions":["Re-quantize the checkpoint with group_size=16 in the quantization_config","If you control the config, set quantization_config['group_size'] = 16 before loading"],"exampleFix":"# before\nquantization_config = {\"quant_method\": \"nvfp4\", \"group_size\": 64}\n# after\nquantization_config = {\"quant_method\": \"nvfp4\", \"group_size\": 16}","handlingStrategy":"validation","validationCode":"assert quantization_config['group_size'] == 16, 'Inkling NVFP4 requires group_size 16'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin group_size=16 in your quantization recipes for Inkling models","Validate the HF quantization_config before launching the server"],"tags":["quantization","nvfp4","config-mismatch","inkling"],"backgroundTag":"quantization-config-invalid","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}