{"record":{"id":"00fe21413d19e3ac","repo":"sgl-project/sglang","slug":"online-mxfp4-requantization-from-compressed-tensor","errorCode":null,"errorMessage":"Online MXFP4 requantization from compressed-tensors NVFP4 checkpoints is not supported at this time.","messagePattern":"Online MXFP4 requantization from compressed-tensors NVFP4 checkpoints is not supported at this time\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/quark/quark.py","lineNumber":116,"sourceCode":"        w0, w1 = weight\n        is_nvfp4_weight = (\n            isinstance(w0, dict)\n            and w0.get(\"dtype\") == \"fp4\"\n            and w0.get(\"qscheme\") == \"per_group\"\n            and w0.get(\"group_size\") == 16\n            and not w0.get(\"is_dynamic\")\n        )\n        is_nvfp4_scale_2 = (\n            isinstance(w1, dict)\n            and w1.get(\"dtype\") == \"fp8_e4m3\"\n            and w1.get(\"qscheme\") == \"per_tensor\"\n            and not w1.get(\"is_dynamic\")\n        )\n        if is_nvfp4_weight and is_nvfp4_scale_2:\n            return Nvfp4SourceConfig()\n        return None\n    if quant_method in (\"compressed-tensors\", \"compressed_tensors\"):\n        raise NotImplementedError(\n            \"Online MXFP4 requantization from compressed-tensors NVFP4 \"\n            \"checkpoints is not supported at this time.\"\n        )\n    return None\n\n\n# Target quant specs used when synthesizing a per-layer config for a\n# MIXED_PRECISION source. The MXFP4 spec is the online-requant target shape\n# recognized by `_is_mx_fp4`; the FP8 spec is the per-tensor W8A8 shape\n# recognized by `_is_fp8_w8a8` (no requantization).\n_MXFP4_TARGET_SPEC: Dict[str, Any] = {\n    \"weight\": {\n        \"dtype\": \"fp4\",\n        \"qscheme\": \"per_group\",\n        \"group_size\": 32,\n        \"is_dynamic\": False,\n        \"scale_format\": \"e8m0\",\n    },","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/quark/quark.py#L98-L134","documentation":"In quark.py, _detect_nvfp4_source inspects the checkpoint's quant config to decide whether online MXFP4 requantization is possible. When quant_method is compressed-tensors/compressed_tensors, the weights may be NVFP4 but packed in a compressed-tensors layout that this online requantization path cannot consume, so it raises NotImplementedError rather than producing garbage.","triggerScenarios":"Loading a checkpoint whose quant config has quant_method \"compressed-tensors\" (or \"compressed_tensors\") with --quantization quark_mxfp4 / requantization into quark MXFP4; from_config -> _detect_nvfp4_source hits the compressed-tensors branch and raises.","commonSituations":"Exporting NVFP4 models via llm-compressor (which writes compressed-tensors configs) and then trying to serve them with sglang's quark_mxfp4 online requantization; assuming all NVFP4 checkpoints are interchangeable across quant frameworks.","solutions":["Serve the compressed-tensors NVFP4 checkpoint with its native method: use the model's own quant config (drop --quantization quark_mxfp4) so sglang's compressed-tensors NVFP4 path handles it.","Or re-export/convert the checkpoint to a quark-style quant config that the online requantization path supports.","Check for a newer sglang version where compressed-tensors NVFP4 online requantization may be supported."],"exampleFix":"# before\npython -m sglang.launch_server --model nvfp4_ct_model --quantization quark_mxfp4  # raises NotImplementedError\n\n# after\npython -m sglang.launch_server --model nvfp4_ct_model  # native compressed-tensors NVFP4 path","handlingStrategy":"validation","validationCode":"qm = config.get(\"quant_method\")\nif qm in (\"compressed-tensors\", \"compressed_tensors\") and requant_to == \"quark_mxfp4\":\n    raise RuntimeError(\"Online MXFP4 requant of compressed-tensors NVFP4 not supported; serve natively\")\nQuarkConfig.from_config(quant_config=config, hf_config=hf_config, ...)","typeGuard":"def supports_online_mxfp4(quant_method: str) -> bool:\n    return quant_method == \"quark\"","tryCatchPattern":"try:\n    cfg = QuarkConfig.from_config(quant_config=config, hf_config=hf_config)\nexcept NotImplementedError as e:\n    if \"compressed-tensors\" in str(e):\n        cfg = None  # fall back to native compressed-tensors loading path\n    else:\n        raise","preventionTips":["Only pass --quantization quark_mxfp4 for quark-format checkpoints.","Automate checkpoint-format detection (quant_method field) in launch scripts.","Track which exporter produced each checkpoint in your model registry."],"tags":["quantization","mxfp4","nvfp4","compressed-tensors","quark","not-implemented","python"],"backgroundTag":"unsupported-checkpoint-format","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}