hiyouga/LlamaFactory · error · ValueError

Per-layer APOLLO does not support gradient accumulation.

Error message

Per-layer APOLLO does not support gradient accumulation.

What it means

Layerwise APOLLO (src/llamafactory/train/trainer_utils.py:338) has the same per-layer immediate-step design as layerwise GaLore, so gradient accumulation is rejected: `apollo_layerwise: true` with `gradient_accumulation_steps != 1` raises ValueError. A prior warning notes displayed grad norms will be zero.

Source

Thrown at src/llamafactory/train/trainer_utils.py:338

        if param.requires_grad:
            trainable_params.append(param)
            if id(param) not in id_apollo_params:
                if name in decay_param_names:
                    decay_params.append(param)
                else:
                    nodecay_params.append(param)

    _, optim_kwargs = Trainer.get_optimizer_cls_and_kwargs(training_args)

    if training_args.optim == "adamw_torch":
        optim_class = APOLLOAdamW
    else:
        raise NotImplementedError(f"Unknown optim: {training_args.optim}.")

    if finetuning_args.apollo_layerwise:
        logger.warning_rank0("The displayed gradient norm will be all zeros in layerwise APOLLO.")
        if training_args.gradient_accumulation_steps != 1:
            raise ValueError("Per-layer APOLLO does not support gradient accumulation.")

        optimizer_dict: dict[torch.Tensor, torch.optim.Optimizer] = {}
        for param in nodecay_params:
            param_groups = [dict(params=[param], weight_decay=0.0)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)
        for param in decay_params:
            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)
        for param in apollo_params:  # apollo params have weight decay
            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay, **apollo_kwargs)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)

        def optimizer_hook(param: "torch.nn.Parameter"):
            if param.grad is not None:
                optimizer_dict[param].step()
                optimizer_dict[param].zero_grad()

        for param in trainable_params:

View on GitHub (pinned to f28afaf635)

Solutions

1. Set `gradient_accumulation_steps: 1` with apollo_layerwise.
2. Compensate effective batch size via per_device_train_batch_size or packing.
3. Turn off `apollo_layerwise` (keep use_apollo) if accumulation is required.

Example fix

# before (yaml)
use_apollo: true
apollo_layerwise: true
gradient_accumulation_steps: 4

# after
use_apollo: true
apollo_layerwise: true
gradient_accumulation_steps: 1

if use_apollo and apollo_layerwise:
    assert gradient_accumulation_steps == 1, "layerwise APOLLO requires gradient_accumulation_steps=1"

When it happens

Trigger: Config with `use_apollo: true`, `apollo_layerwise: true`, and `gradient_accumulation_steps: 2/4/8/...`.

Common situations: Memory-optimized large-model runs combining layerwise APOLLO with accumulation; default YAML templates that already set accumulation > 1.

Related errors

• Per-layer GaLore does not support gradient accumulation.
• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• Cannot use GaLore, APOLLO or BAdam together.
• Distributed training does not support layer-wise APOLLO.
• Please upgrade `transformers` to 4.34.0

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/d5a8115372ce999f. Report an issue: GitHub.