{"record":{"id":"3636feb1ff51ad39","repo":"Lightning-AI/pytorch-lightning","slug":"automatic-gradient-accumulation-and-the-gradienta","errorCode":null,"errorMessage":"Automatic gradient accumulation and the `GradientAccumulationScheduler` is not supported for manual optimization. Please remove the callback or switch to automatic optimization.","messagePattern":"Automatic gradient accumulation and the `GradientAccumulationScheduler` is not supported for manual optimization\\. Please remove the callback or switch to automatic optimization\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py","lineNumber":108,"sourceCode":"        self.epochs = sorted(scheduling.keys())\n\n    def going_to_accumulate_grad_batches(self) -> bool:\n        return any(v > 1 for v in self.scheduling.values())\n\n    def get_accumulate_grad_batches(self, epoch: int) -> int:\n        accumulate_grad_batches = 1\n        for iter_epoch in reversed(self.epochs):\n            if epoch >= iter_epoch:\n                accumulate_grad_batches = self.scheduling[iter_epoch]\n                break\n        return accumulate_grad_batches\n\n    @override\n    def on_train_start(self, trainer: \"pl.Trainer\", pl_module: \"pl.LightningModule\") -> None:\n        \"\"\"Performns a configuration validation before training starts and raises errors for incompatible settings.\"\"\"\n\n        if not pl_module.automatic_optimization:\n            raise RuntimeError(\n                \"\"\"Automatic gradient accumulation and the `GradientAccumulationScheduler` is not supported for\n                manual optimization. Please remove the callback or switch to automatic optimization.\"\"\"\n            )\n\n        overridden_optimizer_step = is_overridden(\"optimizer_step\", pl_module)\n        overridden_optimizer_zero_grad = is_overridden(\"optimizer_zero_grad\", pl_module)\n        going_to_accumulate_grad_batches = self.going_to_accumulate_grad_batches()\n        has_overridden_optimization_functions = overridden_optimizer_step or overridden_optimizer_zero_grad\n        if has_overridden_optimization_functions and going_to_accumulate_grad_batches:\n            rank_zero_warn(\n                \"When using `Trainer(accumulate_grad_batches != 1)` and overriding\"\n                \" `LightningModule.optimizer_{step,zero_grad}`, the hooks will not be called on every batch\"\n                \" (rather, they are called on every optimization step).\"\n            )\n\n        # local import to avoid circular import\n        from lightning.pytorch.strategies import DeepSpeedStrategy\n","sourceCodeStart":90,"sourceCodeEnd":126,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/callbacks/gradient_accumulation_scheduler.py#L90-L126","documentation":"GradientAccumulationScheduler works by intercepting the automatic optimization loop, so it is incompatible with `automatic_optimization=False`. `on_train_start` raises RuntimeError telling you to remove the callback or switch to automatic optimization. With manual optimization you must implement accumulation yourself inside `training_step`/optimizer steps.","triggerScenarios":"`LightningModule.automatic_optimization = False` together with `callbacks=[GradientAccumulationScheduler({0: 4})]` — fails at the start of `trainer.fit`.","commonSituations":"GAN or reinforcement-learning setups using manual optimization where someone adds a gradient accumulation scheduler; migrating a GPT-style manual loop and copying the callback list over.","solutions":["Remove GradientAccumulationScheduler and accumulate manually: call `optimizer.step()/zero_grad()` every N batches in training_step","Or set `automatic_optimization = True` and keep the callback","For schedulers, vary accumulation via your own counter keyed on `self.current_epoch`"],"exampleFix":"# before\nclass LM(LightningModule):\n    automatic_optimization = False\n# Trainer(callbacks=[GradientAccumulationScheduler({0: 8})])\n# after — accumulate manually:\ndef training_step(self, batch, batch_idx):\n    loss = self.step_loss(batch)\n    self.manual_backward(loss)\n    if (batch_idx + 1) % 8 == 0:\n        opt = self.optimizers()\n        opt.step(); opt.zero_grad()","handlingStrategy":"validation","validationCode":"if not model.automatic_optimization:\n    callbacks = [c for c in callbacks if not isinstance(c, GradientAccumulationScheduler)]","typeGuard":"def accumulation_scheduler_ok(pl_module) -> bool:\n    return bool(pl_module.automatic_optimization)","tryCatchPattern":null,"preventionTips":["In manual-optimization modules implement accumulation with a batch counter yourself","Gate callback lists on automatic_optimization in shared trainer factories"],"tags":["lightning","gradient-accumulation","manual-optimization","incompatible-config"],"backgroundTag":"incompatible-training-mode","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}