hiyouga/LlamaFactory · error · ValueError

HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FS

Error message

HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.

What it means

HQQ on-the-fly quantization wraps weights in HQQ parameter types that DeepSpeed ZeRO-3 and FSDP cannot shard/partition. configure_quantization therefore raises ValueError whenever HQQ is requested while either distributed sharding backend is active.

Source

Thrown at src/llamafactory/model/model_utils/quantization.py:211

            # Do not assign device map if:
            # 1. deepspeed zero3 or fsdp (train)
            # 2. auto quantization device map (inference)
            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == "auto":
                if model_args.quantization_bit != 4:
                    raise ValueError("Only 4-bit quantized model can use fsdp+qlora or auto device map.")

                check_version("bitsandbytes>=0.43.0", mandatory=True)
            else:
                init_kwargs["device_map"] = {"": get_current_device()}  # change auto device map for inference

            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.")
        elif model_args.quantization_method == QuantizationMethod.HQQ:
            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:
                raise ValueError("HQQ only accepts 1/2/3/4/5/6/8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():
                raise ValueError("HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.")

            check_version("hqq", mandatory=True)
            init_kwargs["quantization_config"] = HqqConfig(
                nbits=model_args.quantization_bit, quant_zero=False, quant_scale=False, axis=0
            )  # use ATEN kernel (axis=0) for performance
            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with HQQ.")
        elif model_args.quantization_method == QuantizationMethod.EETQ:
            if model_args.quantization_bit != 8:
                raise ValueError("EETQ only accepts 8-bit quantization.")

            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():
                raise ValueError("EETQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.")

            check_version("eetq", mandatory=True)
            init_kwargs["quantization_config"] = EetqConfig()
            logger.info_rank0(f"Quantizing model to {model_args.quantization_bit} bit with EETQ.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Switch quantization_method to bitsandbytes with quantization_bit: 4 — the supported sharded QLoRA path.
2. Or downgrade the deepspeed config to ZeRO-2 (or disable FSDP) when using HQQ.
3. Run HQQ on a single GPU without sharding backends.

Example fix

# before (yaml)
quantization_method: hqq
quantization_bit: 4
deepspeed: examples/deepspeed/ds_z3_config.json

# after (yaml)
quantization_method: bitsandbytes
quantization_bit: 4
deepspeed: examples/deepspeed/ds_z3_config.json

if model_args.quantization_method == "hqq":
    assert not (is_deepspeed_zero3_enabled() or is_fsdp_enabled()), (
        "HQQ is incompatible with ZeRO-3/FSDP; use bnb 4-bit or ZeRO-2"
    )

When it happens

Trigger: quantization_method: hqq (any valid bit) in a training config that also enables deepspeed ZeRO-3 or FSDP (is_deepspeed_zero3_enabled() or is_fsdp_enabled()).

Common situations: Multi-GPU memory-constrained runs trying HQQ + ZeRO-3 to fit a large model; reusing a deepspeed z3 template YAML while experimenting with HQQ.

Related errors

• DeepSpeed ZeRO-3 or FSDP is incompatible with PTQ-quantized
• EETQ quantization is incompatible with DeepSpeed ZeRO-3 or F
• Only 4-bit quantized model can use fsdp+qlora or auto device
• KTransformers is incompatible with DeepSpeed ZeRO-3.
• HQQ only accepts 1/2/3/4/5/6/8-bit quantization.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/d17ac4e498ba6238. Report an issue: GitHub.