{"record":{"id":"3aea106abce5a227","repo":"hiyouga/LlamaFactory","slug":"layer-wise-badam-only-supports-deepspeed-zero-3-tr","errorCode":null,"errorMessage":"Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.","messagePattern":"Layer-wise BAdam only supports DeepSpeed ZeRO-3 training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":516,"sourceCode":"    if finetuning_args.pure_bf16:\n        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):\n            raise ValueError(\"This device does not support `pure_bf16`.\")\n\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.\")\n\n    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:\n        if finetuning_args.use_galore and finetuning_args.galore_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise GaLore.\")\n\n        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise APOLLO.\")\n\n        if finetuning_args.use_badam:\n            if finetuning_args.badam_mode == \"ratio\":\n                raise ValueError(\"Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.\")\n            elif not is_deepspeed_zero3_enabled():\n                raise ValueError(\"Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.\")\n\n    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):\n        raise ValueError(\"GaLore and APOLLO are incompatible with DeepSpeed yet.\")\n\n    if (\n        not finetuning_args.use_mca\n        and not finetuning_args.use_megatron_bridge\n        and training_args.fp8\n        and model_args.quantization_bit is not None\n    ):\n        raise ValueError(\"FP8 training is not compatible with quantization. Please disable one of them.\")\n\n    if model_args.infer_backend != EngineName.HF:\n        raise ValueError(\"vLLM/SGLang backend is only available for API, CLI and Web.\")\n\n    if model_args.use_unsloth and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Unsloth is incompatible with DeepSpeed ZeRO-3.\")\n","sourceCodeStart":498,"sourceCodeEnd":534,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L498-L534","documentation":"Raised in parser.py:516 inside the distributed block when use_badam is true, badam_mode is layer-wise, but DeepSpeed ZeRO-3 is not enabled. Layer-wise BAdam needs the frozen-parameter memory relief of ZeRO-3; under plain DDP/ZeRO-2 the frozen layers' gradients/optimizer overhead would negate its benefit and its implementation relies on ZeRO-3 hooks.","triggerScenarios":"A distributed config with `use_badam: true`, `badam_mode: layer-wise`, but no ZeRO-3 deepspeed config (or a ZeRO stage < 3).","commonSituations":"Following a BAdam example that says 'layer-wise BAdam only supports ZeRO-3' but launching with FORCE_TORCHRUN and no deepspeed file; using DDP by default on a small multi-GPU box.","solutions":["Add a ZeRO-3 DeepSpeed config: `deepspeed: examples/deepspeed/ds_z3_config.json`","Or switch `badam_mode: ratio` only if running single-GPU (ratio is unsupported distributed)","Or disable use_badam and use a conventional optimizer"],"exampleFix":"# before (YAML)\nuse_badam: true\nbadam_mode: layer-wise\n# no deepspeed entry\n\n# after\nuse_badam: true\nbadam_mode: layer-wise\ndeepspeed: examples/deepspeed/ds_z3_config.json","handlingStrategy":"validation","validationCode":"if world_size > 1 and config.get(\"use_badam\") and config.get(\"badam_mode\") == \"layer-wise\" and not is_zero3(config.get(\"deepspeed\")):\n    raise SystemExit(\"layer-wise BAdam requires a ZeRO-3 deepspeed config\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pair badam_mode: layer-wise with ds_z3_config.json in the same template","Check zero stage, not just deepspeed presence"],"tags":["badam","optimizer","deepspeed","zero-3","distributed","incompatible-flags"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}