{"record":{"id":"93dccfb95c3c9638","repo":"Lightning-AI/pytorch-lightning","slug":"currently-model-averaging-cannot-work-with-a-distr","errorCode":null,"errorMessage":"Currently model averaging cannot work with a distributed optimizer of type {optimizer.__class__.__name__}.","messagePattern":"Currently model averaging cannot work with a distributed optimizer of type (.+?)\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/strategies/ddp.py","lineNumber":261,"sourceCode":"                ddp_comm_hook=self._ddp_comm_hook,\n                ddp_comm_wrapper=self._ddp_comm_wrapper,\n            )\n\n    def _enable_model_averaging(self) -> None:\n        log.debug(f\"{self.__class__.__name__}: reinitializing optimizers with post localSGD\")\n        if self._model_averaging_period is None:\n            raise ValueError(\n                \"Post-localSGD algorithm is used, but model averaging period is not provided to DDP strategy.\"\n            )\n        from torch.distributed.optim import DistributedOptimizer, PostLocalSGDOptimizer, ZeroRedundancyOptimizer\n\n        for optimizer in self.optimizers:\n            if isinstance(optimizer, LightningOptimizer):\n                optimizer = optimizer._optimizer\n\n            is_distributed_optimizer = isinstance(optimizer, DistributedOptimizer) if not _IS_WINDOWS else False\n            if isinstance(optimizer, (ZeroRedundancyOptimizer, PostLocalSGDOptimizer)) or is_distributed_optimizer:\n                raise ValueError(\n                    f\"Currently model averaging cannot work with a distributed optimizer of type \"\n                    f\"{optimizer.__class__.__name__}.\"\n                )\n\n        assert self._ddp_comm_state is not None\n        self._model_averager = torch.distributed.algorithms.model_averaging.averagers.PeriodicModelAverager(\n            period=self._model_averaging_period, warmup_steps=self._ddp_comm_state.start_localSGD_iter\n        )\n\n    @override\n    def optimizer_step(\n        self,\n        optimizer: Optimizer,\n        closure: Callable[[], Any],\n        model: Optional[Union[\"pl.LightningModule\", Module]] = None,\n        **kwargs: Any,\n    ) -> Any:\n        \"\"\"Performs the actual optimizer step.","sourceCodeStart":243,"sourceCodeEnd":279,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/strategies/ddp.py#L243-L279","documentation":"Post-localSGD's periodic model averaging is incompatible with optimizers that shard or distributedly update parameters. During setup, _enable_model_averaging inspects self.optimizers and raises ValueError if any is ZeroRedundancyOptimizer, PostLocalSGDOptimizer, a torch DistributedOptimizer (non-Windows), or a LightningOptimizer wrapping one of these.","triggerScenarios":"Combining DDPStrategy(post_local_sgd=True, model_averaging_period=...) with ZeroRedundancyOptimizer (ZeRO-style sharding) or torch.distributed.optim.DistributedOptimizer in your configure_optimizers.","commonSituations":"Trying to stack memory-saving ZeRO/ZeroRedundancy optimizations with post-localSGD communication compression; these algorithms both own parameter synchronization and conflict.","solutions":["Switch to plain torch.optim optimizers (e.g. Adam/SGD) for all optimizers when using post_local_sgd","Or disable post_local_sgd / remove the comm wrapper if you need ZeRO or distributed optimizers","If you intended just gradient communication compression, use a comm hook (e.g. SLOWFAST/FP16 compress) without post-localSGD instead"],"exampleFix":"# before\nclass LitModel(LightningModule):\n    def configure_optimizers(self):\n        return ZeroRedundancyOptimizer(self.parameters(), optimizer_class=torch.optim.Adam, lr=1e-3)\n# + DDPStrategy(post_local_sgd=True, model_averaging_period=5)\n// after\nclass LitModel(LightningModule):\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters(), lr=1e-3)\n# + DDPStrategy(post_local_sgd=True, model_averaging_period=5)","handlingStrategy":"validation","validationCode":"from torch.distributed.optim import ZeroRedundancyOptimizer, DistributedOptimizer\nfrom torch.optim import Optimizer\n\nbad = [o for o in model.configure_optimizers() if not isinstance(o, Optimizer)]\n# ensure no ZeRO/Distributed/PostLocalSGD optimizer is returned when post_local_sgd=True","typeGuard":"def all_plain_optimizers(optimizers) -> bool:\n    return all(isinstance(o, Optimizer) for o in optimizers)","tryCatchPattern":null,"preventionTips":["Don't combine ZeroRedundancyOptimizer with post_local_sgd","Review configure_optimizers before enabling post-localSGD strategies"],"tags":["pytorch","lightning","ddp","distributed-optimizer","incompatibility"],"backgroundTag":"optimizer-strategy-incompatibility","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}