hiyouga/LlamaFactory · error · ValueError

DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized

Error message

DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized models.

What it means

In configure_quantization, when the checkpoint itself carries a quantization_config (PTQ, e.g. GPTQ/AWQ weights), LlamaFactory cannot shard those packed weights across DeepSpeed ZeRO-3 or FSDP processes. MXFP4 and FP8 are exempt because they are dequantized at load; all other PTQ methods raise ValueError when zero3/FSDP is active.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:105

    config: "PretrainedConfig",
    tokenizer: "PreTrainedTokenizer",
    model_args: "ModelArguments",
    is_trainable: bool,
    init_kwargs: dict[str, Any],
) -> None:
    r"""Priority: PTQ-quantized (train/infer) > AutoGPTQ (export) > On-the-fly quantization (train/infer)."""
    if getattr(config, "quantization_config", None):  # ptq
        if model_args.quantization_bit is not None:
            logger.warning_rank0("`quantization_bit` will not affect on the PTQ-quantized models.")

        quantization_config: dict[str, Any] = getattr(config, "quantization_config", None)
        quant_method = quantization_config.get("quant_method", "")

        if quant_method not in (QuantizationMethod.MXFP4, QuantizationMethod.FP8) and (
            is_deepspeed_zero3_enabled() or is_fsdp_enabled()
        ):
            # mxfp4 will dequant the model weights
            raise ValueError("DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized models.")

        if quant_method == QuantizationMethod.MXFP4:
            from transformers import Mxfp4Config

            quant_config = Mxfp4Config(dequantize=True)
            init_kwargs["quantization_config"] = quant_config
            init_kwargs["ignore_mismatched_sizes"] = True

        if quant_method == QuantizationMethod.FP8:
            if _uses_kt_non_expert_cache(model_args):
                if model_args.quantization_bit is not None:
                    raise ValueError("`quantization_bit` cannot be combined with KT weight caches.")

                logger.info_rank0("Skipping source FP8 dequantization because KT weight caches are configured.")
                return

            from transformers import FineGrainedFP8Config

View on GitHub (pinned to f28afaf635)

Solutions

1. Use a plain bf16 base checkpoint and quantize on the fly with quantization_bit/quantization_method (bnb 4-bit) instead of a PTQ checkpoint.
2. Switch the DeepSpeed config from ZeRO-3 to ZeRO-2 (or disable FSDP) for the PTQ model.
3. Prefer an MXFP4 or FP8 PTQ release if you need zero3/FSDP — these are dequantized and shardable.
4. For inference, drop zero3/FSDP entirely and load the PTQ model with a device map.

Example fix

# before
# base model: TheBloke/...-GPTQ + deepspeed zero3 config

# after
# 1) use bf16 base + on-the-fly 4bit:
quantization_bit: 4
quantization_method: bitsandbytes
# 2) or set deepspeed: examples/deepspeed/ds_z2_config.json

qcfg = getattr(config, "quantization_config", None)
if qcfg and qcfg.get("quant_method") not in ("mxfp4", "fp8"):
    assert not (is_deepspeed_zero3_enabled() or is_fsdp_enabled()), (
        "PTQ (GPTQ/AWQ) checkpoints cannot run under ZeRO-3/FSDP; use bf16 base + bnb 4bit"
    )

When it happens

Trigger: Training (or loading under Trainer) a checkpoint with config.quantization_config (GPTQ/AWQ/AQLM etc., quant_method not MXFP4/FP8) while deepspeed zero3 or FSDP is enabled (is_deepspeed_zero3_enabled() or is_fsdp_enabled()).

Common situations: QLoRA-style workflows that mistakenly pair a pre-quantized GPTQ checkpoint with a ZeRO-3 config file; enabling fsdp in the training YAML while the base model is an AWQ release.

Related errors

• HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FS
• EETQ quantization is incompatible with DeepSpeed ZeRO-3 or F
• KTransformers is incompatible with DeepSpeed ZeRO-3.
• Only 4-bit quantized model can use fsdp+qlora or auto device
• vLLM engine does not support bnb quantization (GPTQ and AWQ

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/67a964cb23ad9e8e. Report an issue: GitHub.