hiyouga/LlamaFactory · error · ValueError

GaLore and APOLLO are incompatible with DeepSpeed yet.

Error message

GaLore and APOLLO are incompatible with DeepSpeed yet.

What it means

Raised in parser.py:519 when training_args.deepspeed is not None and either use_galore or use_apollo is true. GaLore and APOLLO replace the standard Adam update with projected low-rank updates, and DeepSpeed's fused optimizers/stages have no integration for them, so the combination is rejected outright (any ZeRO stage).

Source

Thrown at src/llamafactory/hparams/parser.py:519

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

        if finetuning_args.use_badam:
            if finetuning_args.badam_mode == "ratio":
                raise ValueError("Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.")
            elif not is_deepspeed_zero3_enabled():
                raise ValueError("Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.")

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):
        raise ValueError("GaLore and APOLLO are incompatible with DeepSpeed yet.")

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8
        and model_args.quantization_bit is not None
    ):
        raise ValueError("FP8 training is not compatible with quantization. Please disable one of them.")

    if model_args.infer_backend != EngineName.HF:
        raise ValueError("vLLM/SGLang backend is only available for API, CLI and Web.")

    if model_args.use_unsloth and is_deepspeed_zero3_enabled():
        raise ValueError("Unsloth is incompatible with DeepSpeed ZeRO-3.")

    if model_args.use_kt and is_deepspeed_zero3_enabled():
        raise ValueError("KTransformers is incompatible with DeepSpeed ZeRO-3.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Choose one: remove the `deepspeed:` entry and keep GaLore/APOLLO
2. Or keep DeepSpeed and remove use_galore/use_apollo (DeepSpeed ZeRO already reduces optimizer memory)
3. For extreme memory limits, prefer LoRA + ZeRO-2/3 or QLoRA instead of GaLore+DeepSpeed

Example fix

# before (YAML)
deepspeed: examples/deepspeed/ds_z2_config.json
use_galore: true

# after
deepspeed: examples/deepspeed/ds_z2_config.json
# use_galore removed

if config.get("deepspeed") and (config.get("use_galore") or config.get("use_apollo")):
    raise SystemExit("GaLore/APOLLO cannot combine with any DeepSpeed stage")

When it happens

Trigger: Any config with `deepspeed: <config.json>` plus `use_galore: true` or `use_apollo: true`, regardless of zero stage.

Common situations: Trying to stack two memory-saving techniques (ZeRO sharding + GaLore projection) to fine-tune a large model on limited VRAM; enabling GaLore in a template that already carries a deepspeed file.

Related errors

• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• Cannot use GaLore, APOLLO or BAdam together.
• Distributed training does not support layer-wise GaLore.
• Distributed training does not support layer-wise APOLLO.
• Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/0b78e7224872fc0a. Report an issue: GitHub.