hiyouga/LlamaFactory · error · ValueError

EETQ quantization is incompatible with DeepSpeed ZeRO-3 or F

Error message

EETQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.

What it means

Like HQQ, EETQ-quantized parameters cannot be partitioned by DeepSpeed ZeRO-3 or FSDP. configure_quantization raises ValueError when quantization_method: eetq is combined with either sharding backend, before building EetqConfig.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:223

            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.")
        elif model_args.quantization_method == QuantizationMethod.HQQ:
            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:
                raise ValueError("HQQ only accepts 1/2/3/4/5/6/8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():
                raise ValueError("HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.")

            check_version("hqq", mandatory=True)
            init_kwargs["quantization_config"] = HqqConfig(
                nbits=model_args.quantization_bit, quant_zero=False, quant_scale=False, axis=0
            )  # use ATEN kernel (axis=0) for performance
            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with HQQ.")
        elif model_args.quantization_method == QuantizationMethod.EETQ:
            if model_args.quantization_bit != 8:
                raise ValueError("EETQ only accepts 8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():
                raise ValueError("EETQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.")

            check_version("eetq", mandatory=True)
            init_kwargs["quantization_config"] = EetqConfig()
            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with EETQ.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Use bitsandbytes 4-bit (QLoRA) if you need ZeRO-3/FSDP sharding.
2. Or run EETQ with ZeRO-2 / without FSDP on a single process.
3. Precompute a PTQ INT8 checkpoint only if your serving stack supports it; training-side sharding stays unsupported.

Example fix

# before (yaml)
quantization_method: eetq
quantization_bit: 8
deepspeed: examples/deepspeed/ds_z3_config.json

# after (yaml)
quantization_method: eetq
quantization_bit: 8
deepspeed: examples/deepspeed/ds_z2_config.json

if model_args.quantization_method == "eetq":
    assert not (is_deepspeed_zero3_enabled() or is_fsdp_enabled()), (
        "EETQ is incompatible with ZeRO-3/FSDP; use bnb 4-bit or ZeRO-2"
    )

When it happens

Trigger: quantization_method: eetq with quantization_bit: 8 in a run where a ZeRO-3 deepspeed config or FSDP is enabled.

Common situations: Multi-GPU EETQ INT8 attempts with a z3 template; migrating a single-GPU EETQ setup to distributed training unchanged.

Related errors

• DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized
• HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FS
• Only 4-bit quantized model can use fsdp+qlora or auto device
• KTransformers is incompatible with DeepSpeed ZeRO-3.
• EETQ only accepts 8-bit quantization.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/be1c89be6f9433f8. Report an issue: GitHub.