{"record":{"id":"1b8ffdcbf6dea152","repo":"hiyouga/LlamaFactory","slug":"pure-bf16-is-incompatible-with-deepspeed-zero-3","errorCode":null,"errorMessage":"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.","messagePattern":"`pure_bf16` is incompatible with DeepSpeed ZeRO-3\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":503,"sourceCode":"    if training_args.predict_with_generate:\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.\")\n\n        if finetuning_args.compute_accuracy:\n            raise ValueError(\"Cannot use `predict_with_generate` and `compute_accuracy` together.\")\n\n    if training_args.do_train and model_args.quantization_device_map == \"auto\":\n        raise ValueError(\"Cannot use device map for quantized models in training.\")\n\n    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.\")\n\n    if finetuning_args.pure_bf16:\n        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):\n            raise ValueError(\"This device does not support `pure_bf16`.\")\n\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.\")\n\n    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:\n        if finetuning_args.use_galore and finetuning_args.galore_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise GaLore.\")\n\n        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise APOLLO.\")\n\n        if finetuning_args.use_badam:\n            if finetuning_args.badam_mode == \"ratio\":\n                raise ValueError(\"Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.\")\n            elif not is_deepspeed_zero3_enabled():\n                raise ValueError(\"Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.\")\n\n    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):\n        raise ValueError(\"GaLore and APOLLO are incompatible with DeepSpeed yet.\")\n\n    if (","sourceCodeStart":485,"sourceCodeEnd":521,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L485-L521","documentation":"Raised in parser.py:503 when pure_bf16 and DeepSpeed ZeRO-3 are combined. pure_bf16 uses DeepSpeed's bf16 optimizer mode with parameters kept in bf16; ZeRO-3's parameter sharding is not compatible with that pathway in LlamaFactory.","triggerScenarios":"YAML with `pure_bf16: true` plus a deepspeed config at zero stage 3 (ds_z3_config.json).","commonSituations":"Stacking memory tricks: ZeRO-3 for sharding plus pure_bf16 to halve optimizer memory on a bf16 GPU.","solutions":["Drop `pure_bf16: true` and rely on standard bf16: true mixed precision with ZeRO-3","Or keep pure_bf16 but use a ZeRO stage <= 2 deepspeed config"],"exampleFix":"# before (YAML)\ndeepspeed: examples/deepspeed/ds_z3_config.json\npure_bf16: true\n\n# after\ndeepspeed: examples/deepspeed/ds_z3_config.json\nbf16: true\n# pure_bf16 removed","handlingStrategy":"validation","validationCode":"if config.get(\"pure_bf16\") and is_zero3(config.get(\"deepspeed\")):\n    raise SystemExit(\"pure_bf16 + ZeRO-3 unsupported; use bf16: true instead\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer bf16: true unless pure_bf16 is specifically measured to help","Lint configs for the known ZeRO-3 conflict list (pure_bf16, pissa_init, unsloth, kt, predict_with_generate)"],"tags":["pure-bf16","deepspeed","zero-3","incompatible-flags"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}