hiyouga/LlamaFactory · error · ValueError

`pure_bf16` is incompatible with DeepSpeed ZeRO-3.

Error message

`pure_bf16` is incompatible with DeepSpeed ZeRO-3.

What it means

Raised in parser.py:503 when pure_bf16 and DeepSpeed ZeRO-3 are combined. pure_bf16 uses DeepSpeed's bf16 optimizer mode with parameters kept in bf16; ZeRO-3's parameter sharding is not compatible with that pathway in LlamaFactory.

Source

Thrown at src/llamafactory/hparams/parser.py:503

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

        if finetuning_args.use_badam:
            if finetuning_args.badam_mode == "ratio":
                raise ValueError("Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.")
            elif not is_deepspeed_zero3_enabled():
                raise ValueError("Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.")

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):
        raise ValueError("GaLore and APOLLO are incompatible with DeepSpeed yet.")

    if (

View on GitHub (pinned to f28afaf635)

Solutions

1. Drop `pure_bf16: true` and rely on standard bf16: true mixed precision with ZeRO-3
2. Or keep pure_bf16 but use a ZeRO stage <= 2 deepspeed config

Example fix

# before (YAML)
deepspeed: examples/deepspeed/ds_z3_config.json
pure_bf16: true

# after
deepspeed: examples/deepspeed/ds_z3_config.json
bf16: true
# pure_bf16 removed

if config.get("pure_bf16") and is_zero3(config.get("deepspeed")):
    raise SystemExit("pure_bf16 + ZeRO-3 unsupported; use bf16: true instead")

When it happens

Trigger: YAML with `pure_bf16: true` plus a deepspeed config at zero stage 3 (ds_z3_config.json).

Common situations: Stacking memory tricks: ZeRO-3 for sharding plus pure_bf16 to halve optimizer memory on a bf16 GPU.

Related errors

• `predict_with_generate` is incompatible with DeepSpeed ZeRO-
• Please use scripts/pissa_init.py to initialize PiSSA in Deep
• Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.
• Unsloth is incompatible with DeepSpeed ZeRO-3.
• GaLore and APOLLO are incompatible with DeepSpeed yet.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/1b8ffdcbf6dea152. Report an issue: GitHub.