{"record":{"id":"f09f325d9447efc4","repo":"hiyouga/LlamaFactory","slug":"unsloth-is-incompatible-with-deepspeed-zero-3","errorCode":null,"errorMessage":"Unsloth is incompatible with DeepSpeed ZeRO-3.","messagePattern":"Unsloth is incompatible with DeepSpeed ZeRO-3\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":533,"sourceCode":"            elif not is_deepspeed_zero3_enabled():\n                raise ValueError(\"Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.\")\n\n    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):\n        raise ValueError(\"GaLore and APOLLO are incompatible with DeepSpeed yet.\")\n\n    if (\n        not finetuning_args.use_mca\n        and not finetuning_args.use_megatron_bridge\n        and training_args.fp8\n        and model_args.quantization_bit is not None\n    ):\n        raise ValueError(\"FP8 training is not compatible with quantization. Please disable one of them.\")\n\n    if model_args.infer_backend != EngineName.HF:\n        raise ValueError(\"vLLM/SGLang backend is only available for API, CLI and Web.\")\n\n    if model_args.use_unsloth and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Unsloth is incompatible with DeepSpeed ZeRO-3.\")\n\n    if model_args.use_kt and is_deepspeed_zero3_enabled():\n        raise ValueError(\"KTransformers is incompatible with DeepSpeed ZeRO-3.\")\n\n    _set_env_vars()\n    _verify_model_args(model_args, data_args, finetuning_args)\n    _check_extra_dependencies(model_args, finetuning_args, training_args)\n    _verify_trackio_args(training_args)\n\n    if (\n        not finetuning_args.use_mca\n        and not finetuning_args.use_megatron_bridge\n        and training_args.fp8_enable_fsdp_float8_all_gather\n        and not training_args.fp8\n    ):\n        logger.warning_rank0(\"fp8_enable_fsdp_float8_all_gather requires fp8=True. Setting fp8=True.\")\n        model_args.fp8 = True\n","sourceCodeStart":515,"sourceCodeEnd":551,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L515-L551","documentation":"Raised in parser.py:533 when model_args.use_unsloth is true and ZeRO-3 is enabled. Unsloth patches model modules monolithically for speed and expects single-device weight access; ZeRO-3 parameter sharding conflicts with those patches.","triggerScenarios":"Config with `use_unsloth: true` plus a deepspeed config at zero stage 3.","commonSituations":"Turning on Unsloth for faster/lighter LoRA on a config that already used ZeRO-3 to shard a large model across GPUs.","solutions":["Remove `use_unsloth: true` if ZeRO-3 sharding is required for memory","Or switch to a ZeRO stage <= 2 deepspeed config (or none) to keep Unsloth","On a single GPU, just drop the deepspeed entry entirely — Unsloth alone usually suffices"],"exampleFix":"# before (YAML)\nuse_unsloth: true\ndeepspeed: examples/deepspeed/ds_z3_config.json\n\n# after\nuse_unsloth: true\n# deepspeed removed (single-GPU Unsloth run)","handlingStrategy":"validation","validationCode":"if config.get(\"use_unsloth\") and is_zero3(config.get(\"deepspeed\")):\n    raise SystemExit(\"Unsloth conflicts with ZeRO-3; remove one\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["On single GPU use Unsloth without deepspeed; on multi-GPU use Unsloth with ZeRO<=2 or plain DDP","Maintain the ZeRO-3 incompatibility checklist next to your deepspeed configs"],"tags":["unsloth","deepspeed","zero-3","lora","incompatible-flags"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}