{"record":{"id":"1330a573ad6d23d3","repo":"hiyouga/LlamaFactory","slug":"distributed-training-does-not-support-layer-wise-g","errorCode":null,"errorMessage":"Distributed training does not support layer-wise GaLore.","messagePattern":"Distributed training does not support layer-wise GaLore\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":507,"sourceCode":"        if finetuning_args.compute_accuracy:\n            raise ValueError(\"Cannot use `predict_with_generate` and `compute_accuracy` together.\")\n\n    if training_args.do_train and model_args.quantization_device_map == \"auto\":\n        raise ValueError(\"Cannot use device map for quantized models in training.\")\n\n    if finetuning_args.pissa_init and is_deepspeed_zero3_enabled():\n        raise ValueError(\"Please use scripts/pissa_init.py to initialize PiSSA in DeepSpeed ZeRO-3.\")\n\n    if finetuning_args.pure_bf16:\n        if not (is_torch_bf16_gpu_available() or (is_torch_npu_available() and torch.npu.is_bf16_supported())):\n            raise ValueError(\"This device does not support `pure_bf16`.\")\n\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`pure_bf16` is incompatible with DeepSpeed ZeRO-3.\")\n\n    if training_args.parallel_mode == ParallelMode.DISTRIBUTED:\n        if finetuning_args.use_galore and finetuning_args.galore_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise GaLore.\")\n\n        if finetuning_args.use_apollo and finetuning_args.apollo_layerwise:\n            raise ValueError(\"Distributed training does not support layer-wise APOLLO.\")\n\n        if finetuning_args.use_badam:\n            if finetuning_args.badam_mode == \"ratio\":\n                raise ValueError(\"Radio-based BAdam does not yet support distributed training, use layer-wise BAdam.\")\n            elif not is_deepspeed_zero3_enabled():\n                raise ValueError(\"Layer-wise BAdam only supports DeepSpeed ZeRO-3 training.\")\n\n    if training_args.deepspeed is not None and (finetuning_args.use_galore or finetuning_args.use_apollo):\n        raise ValueError(\"GaLore and APOLLO are incompatible with DeepSpeed yet.\")\n\n    if (\n        not finetuning_args.use_mca\n        and not finetuning_args.use_megatron_bridge\n        and training_args.fp8\n        and model_args.quantization_bit is not None","sourceCodeStart":489,"sourceCodeEnd":525,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L489-L525","documentation":"Raised in parser.py:507 inside the ParallelMode.DISTRIBUTED block when use_galore and galore_layerwise are both true. Layer-wise GaLore updates one layer at a time and freezes the rest, which requires all ranks to agree on a synchronized layer schedule that the distributed trainer does not provide.","triggerScenarios":"Multi-GPU launch (torchrun/llamafactory-cli with >1 process) with a config containing `use_galore: true` and `galore_layerwise: true`.","commonSituations":"Taking a single-GPU memory-saving GaLore recipe to a multi-node run; enabling layerwise updates to cut optimizer memory on distributed fine-tunes.","solutions":["Remove `galore_layerwise: true` and use standard GaLore in distributed mode","If layer-wise behavior is required, run single-process (one GPU, no torchrun)","Consider layer-wise BAdam + ZeRO-3 as an alternative layer-wise memory saver that IS supported distributed"],"exampleFix":"# before (YAML)\nuse_galore: true\ngalore_layerwise: true  # multi-GPU launch\n\n# after\nuse_galore: true\n# galore_layerwise removed","handlingStrategy":"validation","validationCode":"if world_size > 1 and config.get(\"use_galore\") and config.get(\"galore_layerwise\"):\n    raise SystemExit(\"galore_layerwise is single-process only\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Condition layerwise optimizer flags on world_size in launch scripts","Keep a single-GPU and a distributed variant of memory-saving configs"],"tags":["galore","optimizer","distributed","layerwise","incompatible-flags"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}