Lightning-AI/pytorch-lightning · warning

It is recommended to use `self.log({result_metric.meta.name!

Error message

It is recommended to use `self.log({result_metric.meta.name!r}, ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.

What it means

PyTorch Lightning emits this warning when a metric logged via self.log(...) is reduced on epoch level (on_epoch=True) while running in a distributed setting (multi-GPU/multi-node) without sync_dist=True. Without sync_dist, each rank computes the metric only over its local shard of data and the epoch value is taken from one rank instead of being correctly accumulated/averaged across devices. The library warns because the resulting epoch-level metric can be silently wrong, not just suboptimal.

Source

Thrown at src/lightning/pytorch/trainer/connectors/logger_connector/result.py:433

        self.update_metrics(key, value, batch_size)

    @torch.compiler.disable
    def update_metrics(self, key: str, value: _VALUE, batch_size: int) -> None:
        result_metric = self[key]
        # performance: avoid calling `__call__` to avoid the checks in `torch.nn.Module._call_impl`
        result_metric.forward(value, batch_size)
        result_metric.has_reset = False

    @staticmethod
    def _get_cache(result_metric: _ResultMetric, on_step: bool) -> Optional[Tensor]:
        cache = None
        if on_step and result_metric.meta.on_step:
            cache = result_metric._forward_cache
        elif not on_step and result_metric.meta.on_epoch:
            if result_metric._computed is None:
                should = result_metric.meta.sync.should
                if not should and result_metric.is_tensor and _distributed_is_initialized():
                    warning_cache.warn(
                        f"It is recommended to use `self.log({result_metric.meta.name!r}, ..., sync_dist=True)`"
                        " when logging on epoch level in distributed setting to accumulate the metric across"
                        " devices.",
                        category=PossibleUserWarning,
                    )
                result_metric.compute()
                result_metric.meta.sync.should = should

            cache = result_metric._computed

        if cache is not None:
            if not isinstance(cache, Tensor):
                raise ValueError(
                    f"The `.compute()` return of the metric logged as {result_metric.meta.name!r} must be a tensor."
                    f" Found {cache}"
                )
            if not result_metric.meta.enable_graph:
                return cache.detach()

View on GitHub (pinned to 9fed5c27d2)

Solutions

Add sync_dist=True to the epoch-level self.log call: self.log('val_loss', loss, on_epoch=True, sync_dist=True).
If you intentionally want rank-local values (rare), silence it explicitly by constructing the warning-free path or filtering PossibleUserWarning, and document why synchronization is unwanted.
Verify your metric semantics: for torchmetrics Metric objects, prefer passing the metric itself and let Lightning aggregate, or set sync_dist with an appropriate sync_dist_op/reduce_fx.
Confirm you actually run distributed: if torch.distributed is not initialized the warning is a false positive caused by a stale environment/world size; check trainer.world_size.

Example fix

// before
self.log("val_loss", loss, on_step=False, on_epoch=True)

// after
self.log("val_loss", loss, on_step=False, on_epoch=True, sync_dist=True)

Defensive patterns

Strategy: validation

Validate before calling

from lightning.pytorch.utilities import rank_zero_only

def log_safe(module, name, value, on_epoch=True):
    sync = module.trainer is not None and module.trainer.world_size > 1
    module.log(name, value, on_epoch=on_epoch, sync_dist=sync)

Prevention

Default to sync_dist=True for every epoch-level metric; it is cheap relative to correct reporting.
Centralize logging in helper methods so the sync_dist policy is applied uniformly across the codebase.
In multi-device tests, assert that logged epoch metrics match single-device runs to catch missing sync_dist early.

When it happens

Trigger: Calling self.log('metric', value, on_step=False, on_epoch=True) (or a Metric object with on_epoch set) without sync_dist=True, while torch.distributed is initialized (ddp, ddp_spawn, deepspeed, fsdp strategies with >1 process). The check fires in _get_cache when the cached metric needs to be computed for the epoch and result_metric.meta.sync.should is False and the value is a tensor.

Common situations: Switching a single-GPU training script to DDP/multi-node without revisiting self.log calls; logging torchmetrics objects that default sync_dist=False; using reduce_fx='mean' on epoch level assuming Lightning already synchronizes; DeepSpeed/FSDP runs where per-rank losses differ and the reported epoch loss looks off.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/2bd8bceac8d33b7f. Report an issue: GitHub.