{"record":{"id":"0b78e7224872fc0a","repo":"hiyouga/LlamaFactory","slug":"galore-and-apollo-are-incompatible-with-deepspeed","errorCode":null,"errorMessage":"GaLore and APOLLO are incompatible with DeepSpeed yet.","messagePattern":"GaLore and APOLLO are incompatible with DeepSpeed yet\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":519,"sourceCode":"\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.\")\n\n    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:\n        if finetuning_args.use_galore and finetuning_args.galore_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise GaLore.\")\n\n        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise APOLLO.\")\n\n        if finetuning_args.use_badam:\n            if finetuning_args.badam_mode == \"ratio\":\n                raise ValueError(\"Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.\")\n            elif not is_deepspeed_zero3_enabled():\n                raise ValueError(\"Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.\")\n\n    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):\n        raise ValueError(\"GaLore and APOLLO are incompatible with DeepSpeed yet.\")\n\n    if (\n        not finetuning_args.use_mca\n        and not finetuning_args.use_megatron_bridge\n        and training_args.fp8\n        and model_args.quantization_bit is not None\n    ):\n        raise ValueError(\"FP8 training is not compatible with quantization. Please disable one of them.\")\n\n    if model_args.infer_backend != EngineName.HF:\n        raise ValueError(\"vLLM/SGLang backend is only available for API, CLI and Web.\")\n\n    if model_args.use_unsloth and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Unsloth is incompatible with DeepSpeed ZeRO-3.\")\n\n    if model_args.use_kt and is_deepspeed_zero3_enabled():\n        raise ValueError(\"KTransformers is incompatible with DeepSpeed ZeRO-3.\")\n","sourceCodeStart":501,"sourceCodeEnd":537,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L501-L537","documentation":"Raised in parser.py:519 when training_args.deepspeed is not None and either use_galore or use_apollo is true. GaLore and APOLLO replace the standard Adam update with projected low-rank updates, and DeepSpeed's fused optimizers/stages have no integration for them, so the combination is rejected outright (any ZeRO stage).","triggerScenarios":"Any config with `deepspeed: <config.json>` plus `use_galore: true` or `use_apollo: true`, regardless of zero stage.","commonSituations":"Trying to stack two memory-saving techniques (ZeRO sharding + GaLore projection) to fine-tune a large model on limited VRAM; enabling GaLore in a template that already carries a deepspeed file.","solutions":["Choose one: remove the `deepspeed:` entry and keep GaLore/APOLLO","Or keep DeepSpeed and remove use_galore/use_apollo (DeepSpeed ZeRO already reduces optimizer memory)","For extreme memory limits, prefer LoRA + ZeRO-2/3 or QLoRA instead of GaLore+DeepSpeed"],"exampleFix":"# before (YAML)\ndeepspeed: examples/deepspeed/ds_z2_config.json\nuse_galore: true\n\n# after\ndeepspeed: examples/deepspeed/ds_z2_config.json\n# use_galore removed","handlingStrategy":"validation","validationCode":"if config.get(\"deepspeed\") and (config.get(\"use_galore\") or config.get(\"use_apollo\")):\n    raise SystemExit(\"GaLore/APOLLO cannot combine with any DeepSpeed stage\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pick exactly one memory-saving strategy per run (DeepSpeed XOR GaLore/APOLLO)","Document the conflict matrix in your team's config cookbook"],"tags":["galore","apollo","deepspeed","optimizer","incompatible-flags"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}