hiyouga/LlamaFactory · error · ValueError

Distributed training does not support layer-wise GaLore.

Error message

Distributed training does not support layer-wise GaLore.

What it means

Raised in parser.py:507 inside the ParallelMode.DISTRIBUTED block when use_galore and galore_layerwise are both true. Layer-wise GaLore updates one layer at a time and freezes the rest, which requires all ranks to agree on a synchronized layer schedule that the distributed trainer does not provide.

Source

Thrown at src/llamafactory/hparams/parser.py:507

        if finetuning_args.compute_accuracy:
            raise ValueError("Cannot use `predict_with_generate` and `compute_accuracy` together.")

    if training_args.do_train and model_args.quantization_device_map == "auto":
        raise ValueError("Cannot use device map for quantized models in training.")

    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():
        raise ValueError("Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.")

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

        if finetuning_args.use_badam:
            if finetuning_args.badam_mode == "ratio":
                raise ValueError("Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.")
            elif not is_deepspeed_zero3_enabled():
                raise ValueError("Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.")

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):
        raise ValueError("GaLore and APOLLO are incompatible with DeepSpeed yet.")

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8
        and model_args.quantization_bit is not None

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove `galore_layerwise: true` and use standard GaLore in distributed mode
2. If layer-wise behavior is required, run single-process (one GPU, no torchrun)
3. Consider layer-wise BAdam + ZeRO-3 as an alternative layer-wise memory saver that IS supported distributed

Example fix

# before (YAML)
use_galore: true
galore_layerwise: true  # multi-GPU launch

# after
use_galore: true
# galore_layerwise removed

if world_size > 1 and config.get("use_galore") and config.get("galore_layerwise"):
    raise SystemExit("galore_layerwise is single-process only")

When it happens

Trigger: Multi-GPU launch (torchrun/llamafactory-cli with >1 process) with a config containing `use_galore: true` and `galore_layerwise: true`.

Common situations: Taking a single-GPU memory-saving GaLore recipe to a multi-node run; enabling layerwise updates to cut optimizer memory on distributed fine-tunes.

Related errors

• Distributed training does not support layer-wise APOLLO.
• Radio-based BAdam does not yet support distributed training,
• Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.
• GaLore and APOLLO are incompatible with DeepSpeed yet.
• Cannot use LoRA with GaLore, APOLLO or BAdam together.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/1330a573ad6d23d3. Report an issue: GitHub.