hiyouga/LlamaFactory · error · ValueError

Per-layer GaLore does not support gradient accumulation.

Error message

Per-layer GaLore does not support gradient accumulation.

What it means

Layerwise GaLore (src/llamafactory/train/trainer_utils.py:251) swaps in a per-parameter optimizer that steps immediately per layer; gradient accumulation across micro-batches is incompatible with that scheme, so `gradient_accumulation_steps != 1` is rejected. The code even logs first that displayed gradient norms will be zero in layerwise mode.

Source

Thrown at src/llamafactory/train/trainer_utils.py:251

                    decay_params.append(param)
                else:
                    nodecay_params.append(param)

    _, optim_kwargs = Trainer.get_optimizer_cls_and_kwargs(training_args)

    if training_args.optim == "adamw_torch":
        optim_class = GaLoreAdamW
    elif training_args.optim in ["adamw_bnb_8bit", "adamw_8bit", "paged_adamw_8bit"]:
        optim_class = GaLoreAdamW8bit
    elif training_args.optim == "adafactor":
        optim_class = GaLoreAdafactor
    else:
        raise NotImplementedError(f"Unknown optim: {training_args.optim}.")

    if finetuning_args.galore_layerwise:
        logger.warning_rank0("The displayed gradient norm will be all zeros in layerwise GaLore.")
        if training_args.gradient_accumulation_steps != 1:
            raise ValueError("Per-layer GaLore does not support gradient accumulation.")

        optimizer_dict: dict[torch.Tensor, torch.optim.Optimizer] = {}
        for param in nodecay_params:
            param_groups = [dict(params=[param], weight_decay=0.0)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)
        for param in decay_params:
            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)
        for param in galore_params:  # galore params have weight decay
            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay, **galore_kwargs)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)

        def optimizer_hook(param: "torch.nn.Parameter"):
            if param.grad is not None:
                optimizer_dict[param].step()
                optimizer_dict[param].zero_grad()

        for param in trainable_params:

View on GitHub (pinned to f28afaf635)

Solutions

1. Set `gradient_accumulation_steps: 1` when using galore_layerwise.
2. To keep the effective batch size, increase `per_device_train_batch_size` instead, or enable packing.
3. If accumulation is mandatory, disable `galore_layerwise` and use standard (projected) GaLore.

Example fix

# before (yaml)
use_galore: true
galore_layerwise: true
gradient_accumulation_steps: 8

# after
use_galore: true
galore_layerwise: true
gradient_accumulation_steps: 1
per_device_train_batch_size: 8

if use_galore and galore_layerwise:
    assert gradient_accumulation_steps == 1, "layerwise GaLore requires gradient_accumulation_steps=1"

When it happens

Trigger: Config with `use_galore: true`, `galore_layerwise: true`, and `gradient_accumulation_steps: > 1` (2, 4, 8 are common defaults for large effective batch sizes).

Common situations: Users simulating a large batch with accumulation on limited VRAM while also enabling layerwise GaLore to save memory; template configs that ship with gradient_accumulation_steps: 8.

Related errors

• Per-layer APOLLO does not support gradient accumulation.
• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• Cannot use GaLore, APOLLO or BAdam together.
• Distributed training does not support layer-wise GaLore.
• Unknown optim: {training_args.optim}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/d21bdf5a434ba69d. Report an issue: GitHub.