{"record":{"id":"d17ac4e498ba6238","repo":"hiyouga/LlamaFactory","slug":"hqq-quantization-is-incompatible-with-deepspeed-ze","errorCode":null,"errorMessage":"HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.","messagePattern":"HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/quantization.py","lineNumber":211,"sourceCode":"\n            # Do not assign device map if:\n            # 1. deepspeed zero3 or fsdp (train)\n            # 2. auto quantization device map (inference)\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled() or model_args.quantization_device_map == \"auto\":\n                if model_args.quantization_bit != 4:\n                    raise ValueError(\"Only 4-bit quantized model can use fsdp+qlora or auto device map.\")\n\n                check_version(\"bitsandbytes>=0.43.0\", mandatory=True)\n            else:\n                init_kwargs[\"device_map\"] = {\"\": get_current_device()}  # change auto device map for inference\n\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with bitsandbytes.\")\n        elif model_args.quantization_method == QuantizationMethod.HQQ:\n            if model_args.quantization_bit not in [8, 6, 5, 4, 3, 2, 1]:\n                raise ValueError(\"HQQ only accepts 1/2/3/4/5/6/8-bit quantization.\")\n\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():\n                raise ValueError(\"HQQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.\")\n\n            check_version(\"hqq\", mandatory=True)\n            init_kwargs[\"quantization_config\"] = HqqConfig(\n                nbits=model_args.quantization_bit, quant_zero=False, quant_scale=False, axis=0\n            )  # use ATEN kernel (axis=0) for performance\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with HQQ.\")\n        elif model_args.quantization_method == QuantizationMethod.EETQ:\n            if model_args.quantization_bit != 8:\n                raise ValueError(\"EETQ only accepts 8-bit quantization.\")\n\n            if is_deepspeed_zero3_enabled() or is_fsdp_enabled():\n                raise ValueError(\"EETQ quantization is incompatible with DeepSpeed ZeRO-3 or FSDP.\")\n\n            check_version(\"eetq\", mandatory=True)\n            init_kwargs[\"quantization_config\"] = EetqConfig()\n            logger.info_rank0(f\"Quantizing model to {model_args.quantization_bit} bit with EETQ.\")\n","sourceCodeStart":193,"sourceCodeEnd":228,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/quantization.py#L193-L228","documentation":"HQQ on-the-fly quantization wraps weights in HQQ parameter types that DeepSpeed ZeRO-3 and FSDP cannot shard/partition. configure_quantization therefore raises ValueError whenever HQQ is requested while either distributed sharding backend is active.","triggerScenarios":"quantization_method: hqq (any valid bit) in a training config that also enables deepspeed ZeRO-3 or FSDP (is_deepspeed_zero3_enabled() or is_fsdp_enabled()).","commonSituations":"Multi-GPU memory-constrained runs trying HQQ + ZeRO-3 to fit a large model; reusing a deepspeed z3 template YAML while experimenting with HQQ.","solutions":["Switch quantization_method to bitsandbytes with quantization_bit: 4 — the supported sharded QLoRA path.","Or downgrade the deepspeed config to ZeRO-2 (or disable FSDP) when using HQQ.","Run HQQ on a single GPU without sharding backends."],"exampleFix":"# before (yaml)\nquantization_method: hqq\nquantization_bit: 4\ndeepspeed: examples/deepspeed/ds_z3_config.json\n\n# after (yaml)\nquantization_method: bitsandbytes\nquantization_bit: 4\ndeepspeed: examples/deepspeed/ds_z3_config.json","handlingStrategy":"validation","validationCode":"if model_args.quantization_method == \"hqq\":\n    assert not (is_deepspeed_zero3_enabled() or is_fsdp_enabled()), (\n        \"HQQ is incompatible with ZeRO-3/FSDP; use bnb 4-bit or ZeRO-2\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat hqq and eetq as single-process / non-sharded methods.","For multi-GPU quantized training, default to bitsandbytes 4-bit QLoRA."],"tags":["hqq","quantization","deepspeed","fsdp","zero3"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}