hiyouga/LlamaFactory · error · ValueError

Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.

Error message

Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.

What it means

Raised in parser.py:516 inside the distributed block when use_badam is true, badam_mode is layer-wise, but DeepSpeed ZeRO-3 is not enabled. Layer-wise BAdam needs the frozen-parameter memory relief of ZeRO-3; under plain DDP/ZeRO-2 the frozen layers' gradients/optimizer overhead would negate its benefit and its implementation relies on ZeRO-3 hooks.

Source

Thrown at src/llamafactory/hparams/parser.py:516

    if finetuning_args.pure_bf16:
        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):
            raise ValueError("This device does not support `pure_bf16`.")

        if is_deepspeed_zero3_enabled():
            raise ValueError("`pure_bf16` is incompatible with DeepSpeed ZeRO-3.")

    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:
        if finetuning_args.use_galore and finetuning_args.galore_layerwise:
            raise ValueError("Distributed training does not support layer-wise GaLore.")

        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:
            raise ValueError("Distributed training does not support layer-wise APOLLO.")

        if finetuning_args.use_badam:
            if finetuning_args.badam_mode == "ratio":
                raise ValueError("Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.")
            elif not is_deepspeed_zero3_enabled():
                raise ValueError("Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.")

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):
        raise ValueError("GaLore and APOLLO are incompatible with DeepSpeed yet.")

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8
        and model_args.quantization_bit is not None
    ):
        raise ValueError("FP8 training is not compatible with quantization. Please disable one of them.")

    if model_args.infer_backend != EngineName.HF:
        raise ValueError("vLLM/SGLang backend is only available for API, CLI and Web.")

    if model_args.use_unsloth and is_deepspeed_zero3_enabled():
        raise ValueError("Unsloth is incompatible with DeepSpeed ZeRO-3.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Add a ZeRO-3 DeepSpeed config: `deepspeed: examples/deepspeed/ds_z3_config.json`
2. Or switch `badam_mode: ratio` only if running single-GPU (ratio is unsupported distributed)
3. Or disable use_badam and use a conventional optimizer

Example fix

# before (YAML)
use_badam: true
badam_mode: layer-wise
# no deepspeed entry

# after
use_badam: true
badam_mode: layer-wise
deepspeed: examples/deepspeed/ds_z3_config.json

if world_size > 1 and config.get("use_badam") and config.get("badam_mode") == "layer-wise" and not is_zero3(config.get("deepspeed")):
    raise SystemExit("layer-wise BAdam requires a ZeRO-3 deepspeed config")

When it happens

Trigger: A distributed config with `use_badam: true`, `badam_mode: layer-wise`, but no ZeRO-3 deepspeed config (or a ZeRO stage < 3).

Common situations: Following a BAdam example that says 'layer-wise BAdam only supports ZeRO-3' but launching with FORCE_TORCHRUN and no deepspeed file; using DDP by default on a small multi-GPU box.

Related errors

• Radio-based BAdam does not yet support distributed training,
• `predict_with_generate` is incompatible with DeepSpeed ZeRO-
• Please use scripts/pissa_init.py to initialize PiSSA in Deep
• `pure_bf16` is incompatible with DeepSpeed ZeRO-3.
• Distributed training does not support layer-wise GaLore.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/3aea106abce5a227. Report an issue: GitHub.