Lightning-AI/pytorch-lightning · error · RuntimeError

Automatic gradient accumulation and the `GradientAccumulatio

Error message

Automatic gradient accumulation and the `GradientAccumulationScheduler` is not supported for manual optimization. Please remove the callback or switch to automatic optimization.

What it means

GradientAccumulationScheduler works by intercepting the automatic optimization loop, so it is incompatible with `automatic_optimization=False`. `on_train_start` raises RuntimeError telling you to remove the callback or switch to automatic optimization. With manual optimization you must implement accumulation yourself inside `training_step`/optimizer steps.

Source

Thrown at src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py:108

        self.epochs = sorted(scheduling.keys())

    def going_to_accumulate_grad_batches(self) -> bool:
        return any(v > 1 for v in self.scheduling.values())

    def get_accumulate_grad_batches(self, epoch: int) -> int:
        accumulate_grad_batches = 1
        for iter_epoch in reversed(self.epochs):
            if epoch >= iter_epoch:
                accumulate_grad_batches = self.scheduling[iter_epoch]
                break
        return accumulate_grad_batches

    @override
    def on_train_start(self, trainer: "pl.Trainer", pl_module: "pl.LightningModule") -> None:
        """Performns a configuration validation before training starts and raises errors for incompatible settings."""

        if not pl_module.automatic_optimization:
            raise RuntimeError(
                """Automatic gradient accumulation and the `GradientAccumulationScheduler` is not supported for
                manual optimization. Please remove the callback or switch to automatic optimization."""
            )

        overridden_optimizer_step = is_overridden("optimizer_step", pl_module)
        overridden_optimizer_zero_grad = is_overridden("optimizer_zero_grad", pl_module)
        going_to_accumulate_grad_batches = self.going_to_accumulate_grad_batches()
        has_overridden_optimization_functions = overridden_optimizer_step or overridden_optimizer_zero_grad
        if has_overridden_optimization_functions and going_to_accumulate_grad_batches:
            rank_zero_warn(
                "When using `Trainer(accumulate_grad_batches != 1)` and overriding"
                " `LightningModule.optimizer_{step,zero_grad}`, the hooks will not be called on every batch"
                " (rather, they are called on every optimization step)."
            )

        # local import to avoid circular import
        from lightning.pytorch.strategies import DeepSpeedStrategy

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Remove GradientAccumulationScheduler and accumulate manually: call `optimizer.step()/zero_grad()` every N batches in training_step
2. Or set `automatic_optimization = True` and keep the callback
3. For schedulers, vary accumulation via your own counter keyed on `self.current_epoch`

Example fix

# before
class LM(LightningModule):
    automatic_optimization = False
# Trainer(callbacks=[GradientAccumulationScheduler({0: 8})])
# after — accumulate manually:
def training_step(self, batch, batch_idx):
    loss = self.step_loss(batch)
    self.manual_backward(loss)
    if (batch_idx + 1) % 8 == 0:
        opt = self.optimizers()
        opt.step(); opt.zero_grad()

if not model.automatic_optimization:
    callbacks = [c for c in callbacks if not isinstance(c, GradientAccumulationScheduler)]

def accumulation_scheduler_ok(pl_module) -> bool:
    return bool(pl_module.automatic_optimization)

When it happens

Trigger: `LightningModule.automatic_optimization = False` together with `callbacks=[GradientAccumulationScheduler({0: 4})]` — fails at the start of `trainer.fit`.

Common situations: GAN or reinforcement-learning setups using manual optimization where someone adds a gradient accumulation scheduler; migrating a GPT-style manual loop and copying the callback list over.

Related errors

• Empty dict cannot be interpreted correct
• Epoch should be an int greater than or equal to 0. Got {list
• Accumulation factor should be an int greater than 0. Got {li
• Epochs indexing from 1, epoch {minimal_epoch} cannot be inte
• The `{type(trainer.strategy).__name__}` does not support `ac

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/3636feb1ff51ad39. Report an issue: GitHub.