hiyouga/LlamaFactory · error · ValueError

Distributed training does not support layer-wise APOLLO.

Error message

Distributed training does not support layer-wise APOLLO.

What it means

Raised in parser.py:510 inside the distributed block when use_apollo and apollo_layerwise are both true. APOLLO is a GaLore-like low-rank projector optimizer; its layer-wise variant has the same cross-rank synchronization problem as layer-wise GaLore, so it is rejected in distributed runs.

Source

Thrown at src/llamafactory/hparams/parser.py:510

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

        if finetuning_args.use_badam:
            if finetuning_args.badam_mode == "ratio":
                raise ValueError("Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.")
            elif not is_deepspeed_zero3_enabled():
                raise ValueError("Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.")

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):
        raise ValueError("GaLore and APOLLO are incompatible with DeepSpeed yet.")

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8
        and model_args.quantization_bit is not None
    ):
        raise ValueError("FP8 training is not compatible with quantization. Please disable one of them.")

View on GitHub (pinned to f28afaf635)

Solutions

Remove `apollo_layerwise: true` for distributed training
If layer-wise is required, run on a single process/GPU without torchrun

Example fix

# before (YAML)
use_apollo: true
apollo_layerwise: true  # multi-GPU launch

# after
use_apollo: true
# apollo_layerwise removed

Defensive patterns

Strategy: validation

Validate before calling

if world_size > 1 and config.get("use_apollo") and config.get("apollo_layerwise"):
    raise SystemExit("apollo_layerwise is single-process only")

Prevention

Apply the same distributed lint to APOLLO flags as to GaLore
Avoid porting single-GPU layerwise recipes to torchrun unchanged

When it happens

Trigger: torchrun/llamafactory-cli multi-process launch with `use_apollo: true` and `apollo_layerwise: true` in the config.

Common situations: Scaling a single-GPU APOLLO memory-saving recipe to multiple GPUs; mixing APOLLO layerwise with DeepSpeed or DDP setups.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/ac5fb997441c906e. Report an issue: GitHub.