Lightning-AI/pytorch-lightning · error · RuntimeError

The `{type(trainer.strategy).__name__}` does not support `ac

Error message

The `{type(trainer.strategy).__name__}` does not support `accumulate_grad_batches` changing between epochs.

What it means

DeepSpeed bakes gradient accumulation into its engine config at initialization, so changing `accumulate_grad_batches` between epochs (which GradientAccumulationScheduler does) cannot be supported. `on_train_start` raises RuntimeError naming the strategy class when it is DeepSpeedStrategy.

Source

Thrown at src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py:128

                manual optimization. Please remove the callback or switch to automatic optimization."""
            )

        overridden_optimizer_step = is_overridden("optimizer_step", pl_module)
        overridden_optimizer_zero_grad = is_overridden("optimizer_zero_grad", pl_module)
        going_to_accumulate_grad_batches = self.going_to_accumulate_grad_batches()
        has_overridden_optimization_functions = overridden_optimizer_step or overridden_optimizer_zero_grad
        if has_overridden_optimization_functions and going_to_accumulate_grad_batches:
            rank_zero_warn(
                "When using `Trainer(accumulate_grad_batches != 1)` and overriding"
                " `LightningModule.optimizer_{step,zero_grad}`, the hooks will not be called on every batch"
                " (rather, they are called on every optimization step)."
            )

        # local import to avoid circular import
        from lightning.pytorch.strategies import DeepSpeedStrategy

        if isinstance(trainer.strategy, DeepSpeedStrategy):
            raise RuntimeError(
                f"The `{type(trainer.strategy).__name__}` does not support `accumulate_grad_batches` changing"
                " between epochs."
            )
        if trainer.accumulate_grad_batches != 1:
            raise ValueError(
                "You have set `accumulate_grad_batches` and are using the `GradientAccumulationScheduler`"
                " callback. Either remove `accumulate_grad_batches` from the Trainer or remove the callback."
            )

    @override
    def on_train_epoch_start(self, trainer: "pl.Trainer", *_: Any) -> None:
        trainer.accumulate_grad_batches = self.get_accumulate_grad_batches(trainer.current_epoch)

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Use DeepSpeed's fixed `accumulate_grad_batches` on the Trainer and remove GradientAccumulationScheduler
2. For dynamic schedules, restart training stages with separate Trainer runs using different fixed accumulation values
3. If a per-epoch schedule is essential, use a non-DeepSpeed strategy

Example fix

# before
Trainer(strategy='deepspeed', callbacks=[GradientAccumulationScheduler({0: 1, 5: 4})])
# after
Trainer(strategy='deepspeed', accumulate_grad_batches=4)

from lightning.pytorch.strategies import DeepSpeedStrategy
if isinstance(trainer.strategy, DeepSpeedStrategy):
    assert not any(isinstance(c, GradientAccumulationScheduler) for c in callbacks)

def scheduler_compatible(trainer) -> bool:
    from lightning.pytorch.strategies import DeepSpeedStrategy
    return not isinstance(trainer.strategy, DeepSpeedStrategy)

When it happens

Trigger: `Trainer(strategy='deepspeed', accumulate_grad_batches=..., callbacks=[GradientAccumulationScheduler(...)])` — fails right at train start.

Common situations: Adding an epoch-based accumulation schedule for a warmup/curriculum while using DeepSpeed ZeRO; migrating from DDP to DeepSpeed and keeping the callback.

Related errors

• The Finetuning callback does not support running with the De
• No models were set up for backward. Did you forget to call `
• When using multiple models + deepspeed, please provide the m
• The `{type(self._strategy).__name__}` requires the model and
• Empty dict cannot be interpreted correct

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/50bbe6cea4665b7a. Report an issue: GitHub.