Lightning-AI/pytorch-lightning · error · ValueError

Post-localSGD algorithm is used, but model averaging period

Error message

Post-localSGD algorithm is used, but model averaging period is not provided to DDP strategy.

What it means

When the DDP strategy is configured with post-localSGD (a communication hook enabling local SGD with periodic averaging), you must also pass model_averaging_period. _enable_model_averaging raises ValueError during setup if the period is missing, because PeriodicModelAverager cannot be constructed without it.

Source

Thrown at src/lightning/pytorch/strategies/ddp.py:250

        rank_zero_only.rank = utils_rank_zero_only.rank = self.global_rank

    def _register_ddp_hooks(self) -> None:
        log.debug(f"{self.__class__.__name__}: registering ddp hooks")
        # currently, DDP communication hooks only work with NCCL backend and SPSD (single process single device) mode
        # https://github.com/pytorch/pytorch/blob/v1.8.0/torch/nn/parallel/distributed.py#L1080-L1084
        if self.root_device.type == "cuda":
            assert isinstance(self.model, DistributedDataParallel)
            _register_ddp_comm_hook(
                model=self.model,
                ddp_comm_state=self._ddp_comm_state,
                ddp_comm_hook=self._ddp_comm_hook,
                ddp_comm_wrapper=self._ddp_comm_wrapper,
            )

    def _enable_model_averaging(self) -> None:
        log.debug(f"{self.__class__.__name__}: reinitializing optimizers with post localSGD")
        if self._model_averaging_period is None:
            raise ValueError(
                "Post-localSGD algorithm is used, but model averaging period is not provided to DDP strategy."
            )
        from torch.distributed.optim import DistributedOptimizer, PostLocalSGDOptimizer, ZeroRedundancyOptimizer

        for optimizer in self.optimizers:
            if isinstance(optimizer, LightningOptimizer):
                optimizer = optimizer._optimizer

            is_distributed_optimizer = isinstance(optimizer, DistributedOptimizer) if not _IS_WINDOWS else False
            if isinstance(optimizer, (ZeroRedundancyOptimizer, PostLocalSGDOptimizer)) or is_distributed_optimizer:
                raise ValueError(
                    f"Currently model averaging cannot work with a distributed optimizer of type "
                    f"{optimizer.__class__.__name__}."
                )

        assert self._ddp_comm_state is not None
        self._model_averager = torch.distributed.algorithms.model_averaging.averagers.PeriodicModelAverager(
            period=self._model_averaging_period, warmup_steps=self._ddp_comm_state.start_localSGD_iter

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Pass the averaging interval, e.g. DDPStrategy(post_local_sgd=True, model_averaging_period=10) (steps between synchronizations)
2. Or drop post_local_sgd=True if periodic model averaging is not intended

Example fix

# before
strategy = DDPStrategy(post_local_sgd=True)
// after
strategy = DDPStrategy(post_local_sgd=True, model_averaging_period=10)

strategy = DDPStrategy(post_local_sgd=True)
assert strategy._model_averaging_period is not None, \
    "post_local_sgd requires model_averaging_period"

When it happens

Trigger: Setting DDPStrategy(post_local_sgd=True) (or an equivalent comm hook config) without providing model_averaging_period to the strategy.

Common situations: Enabling post-localSGD after reading a tutorial that omitted the period argument; copying a strategy config where the period was set via a different parameter name across Lightning versions.

Related errors

• To use Fabric with more than one device, you must call `.lau
• Currently model averaging cannot work with a distributed opt
• `num_nodes` must be a positive integer, but got {num_nodes}.
• You need to set up the model first before you can call `fabr
• Trying to inject a modified sampler into the batch sampler;

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/aad5029d46d054af. Report an issue: GitHub.