hiyouga/LlamaFactory · error · ValueError

Unsloth is incompatible with DeepSpeed ZeRO-3.

Error message

Unsloth is incompatible with DeepSpeed ZeRO-3.

What it means

Raised in parser.py:533 when model_args.use_unsloth is true and ZeRO-3 is enabled. Unsloth patches model modules monolithically for speed and expects single-device weight access; ZeRO-3 parameter sharding conflicts with those patches.

Source

Thrown at src/llamafactory/hparams/parser.py:533

            elif not is_deepspeed_zero3_enabled():
                raise ValueError("Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.")

    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):
        raise ValueError("GaLore and APOLLO are incompatible with DeepSpeed yet.")

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8
        and model_args.quantization_bit is not None
    ):
        raise ValueError("FP8 training is not compatible with quantization. Please disable one of them.")

    if model_args.infer_backend != EngineName.HF:
        raise ValueError("vLLM/SGLang backend is only available for API, CLI and Web.")

    if model_args.use_unsloth and is_deepspeed_zero3_enabled():
        raise ValueError("Unsloth is incompatible with DeepSpeed ZeRO-3.")

    if model_args.use_kt and is_deepspeed_zero3_enabled():
        raise ValueError("KTransformers is incompatible with DeepSpeed ZeRO-3.")

    _set_env_vars()
    _verify_model_args(model_args, data_args, finetuning_args)
    _check_extra_dependencies(model_args, finetuning_args, training_args)
    _verify_trackio_args(training_args)

    if (
        not finetuning_args.use_mca
        and not finetuning_args.use_megatron_bridge
        and training_args.fp8_enable_fsdp_float8_all_gather
        and not training_args.fp8
    ):
        logger.warning_rank0("fp8_enable_fsdp_float8_all_gather requires fp8=True. Setting fp8=True.")
        model_args.fp8 = True

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove `use_unsloth: true` if ZeRO-3 sharding is required for memory
2. Or switch to a ZeRO stage <= 2 deepspeed config (or none) to keep Unsloth
3. On a single GPU, just drop the deepspeed entry entirely — Unsloth alone usually suffices

Example fix

# before (YAML)
use_unsloth: true
deepspeed: examples/deepspeed/ds_z3_config.json

# after
use_unsloth: true
# deepspeed removed (single-GPU Unsloth run)

if config.get("use_unsloth") and is_zero3(config.get("deepspeed")):
    raise SystemExit("Unsloth conflicts with ZeRO-3; remove one")

When it happens

Trigger: Config with `use_unsloth: true` plus a deepspeed config at zero stage 3.

Common situations: Turning on Unsloth for faster/lighter LoRA on a config that already used ZeRO-3 to shard a large model across GPUs.

Related errors

• Please use scripts/pissa_init.py to initialize PiSSA in Deep
• `predict_with_generate` is incompatible with DeepSpeed ZeRO-
• `pure_bf16` is incompatible with DeepSpeed ZeRO-3.
• Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.
• Unsloth does not support lora reward model.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f09f325d9447efc4. Report an issue: GitHub.