{"record":{"id":"67a964cb23ad9e8e","repo":"hiyouga/LlamaFactory","slug":"deepspeed-zero-3-or-fsdp-is-incompatible-with-ptq","errorCode":null,"errorMessage":"DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized models.","messagePattern":"DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized models\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/quantization.py","lineNumber":105,"sourceCode":"    config: \"PretrainedConfig\",\n    tokenizer: \"PreTrainedTokenizer\",\n    model_args: \"ModelArguments\",\n    is_trainable: bool,\n    init_kwargs: dict[str, Any],\n) -> None:\n    r\"\"\"Priority: PTQ-quantized (train/infer) > AutoGPTQ (export) > On-the-fly quantization (train/infer).\"\"\"\n    if getattr(config, \"quantization_config\", None):  # ptq\n        if model_args.quantization_bit is not None:\n            logger.warning_rank0(\"`quantization_bit` will not affect on the PTQ-quantized models.\")\n\n        quantization_config: dict[str, Any] = getattr(config, \"quantization_config\", None)\n        quant_method = quantization_config.get(\"quant_method\", \"\")\n\n        if quant_method not in (QuantizationMethod.MXFP4, QuantizationMethod.FP8) and (\n            is_deepspeed_zero3_enabled() or is_fsdp_enabled()\n        ):\n            # mxfp4 will dequant the model weights\n            raise ValueError(\"DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized models.\")\n\n        if quant_method == QuantizationMethod.MXFP4:\n            from transformers import Mxfp4Config\n\n            quant_config = Mxfp4Config(dequantize=True)\n            init_kwargs[\"quantization_config\"] = quant_config\n            init_kwargs[\"ignore_mismatched_sizes\"] = True\n\n        if quant_method == QuantizationMethod.FP8:\n            if _uses_kt_non_expert_cache(model_args):\n                if model_args.quantization_bit is not None:\n                    raise ValueError(\"`quantization_bit` cannot be combined with KT weight caches.\")\n\n                logger.info_rank0(\"Skipping source FP8 dequantization because KT weight caches are configured.\")\n                return\n\n            from transformers import FineGrainedFP8Config\n","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/quantization.py#L87-L123","documentation":"In configure_quantization, when the checkpoint itself carries a quantization_config (PTQ, e.g. GPTQ/AWQ weights), LlamaFactory cannot shard those packed weights across DeepSpeed ZeRO-3 or FSDP processes. MXFP4 and FP8 are exempt because they are dequantized at load; all other PTQ methods raise ValueError when zero3/FSDP is active.","triggerScenarios":"Training (or loading under Trainer) a checkpoint with config.quantization_config (GPTQ/AWQ/AQLM etc., quant_method not MXFP4/FP8) while deepspeed zero3 or FSDP is enabled (is_deepspeed_zero3_enabled() or is_fsdp_enabled()).","commonSituations":"QLoRA-style workflows that mistakenly pair a pre-quantized GPTQ checkpoint with a ZeRO-3 config file; enabling fsdp in the training YAML while the base model is an AWQ release.","solutions":["Use a plain bf16 base checkpoint and quantize on the fly with quantization_bit/quantization_method (bnb 4-bit) instead of a PTQ checkpoint.","Switch the DeepSpeed config from ZeRO-3 to ZeRO-2 (or disable FSDP) for the PTQ model.","Prefer an MXFP4 or FP8 PTQ release if you need zero3/FSDP — these are dequantized and shardable.","For inference, drop zero3/FSDP entirely and load the PTQ model with a device map."],"exampleFix":"# before\n# base model: TheBloke/...-GPTQ + deepspeed zero3 config\n\n# after\n# 1) use bf16 base + on-the-fly 4bit:\nquantization_bit: 4\nquantization_method: bitsandbytes\n# 2) or set deepspeed: examples/deepspeed/ds_z2_config.json","handlingStrategy":"validation","validationCode":"qcfg = getattr(config, \"quantization_config\", None)\nif qcfg and qcfg.get(\"quant_method\") not in (\"mxfp4\", \"fp8\"):\n    assert not (is_deepspeed_zero3_enabled() or is_fsdp_enabled()), (\n        \"PTQ (GPTQ/AWQ) checkpoints cannot run under ZeRO-3/FSDP; use bf16 base + bnb 4bit\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never pair pre-quantized GPTQ/AWQ checkpoints with z3/FSDP deepspeed configs.","Standardize QLoRA recipe: unquantized base + quantization_bit: 4 + bitsandbytes."],"tags":["quantization","gptq","deepspeed","fsdp","zero3","training-config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}