Lightning-AI/pytorch-lightning · error · RuntimeError

Accessing the device mesh before processes have initialized

Error message

Accessing the device mesh before processes have initialized is not allowed.

What it means

ModelParallelStrategy lazily creates its DeviceMesh during environment setup (setup_environment). The device_mesh property raises if accessed before that setup ran, because no process group or mesh exists yet. Any code touching strategy.device_mesh (or attributes deriving from it) prior to Fabric's setup phase triggers this error.

Source

Thrown at src/lightning/fabric/strategies/model_parallel.py:120

        process_group_backend: Optional[str] = None,
        timeout: Optional[timedelta] = default_pg_timeout,
    ) -> None:
        super().__init__()
        self._parallelize_fn = parallelize_fn
        self._data_parallel_size = data_parallel_size
        self._tensor_parallel_size = tensor_parallel_size
        self._num_nodes = 1
        self._save_distributed_checkpoint = save_distributed_checkpoint
        self._process_group_backend: Optional[str] = process_group_backend
        self._timeout: Optional[timedelta] = timeout
        self._backward_sync_control = _ParallelBackwardSyncControl()

        self._device_mesh: Optional[DeviceMesh] = None

    @property
    def device_mesh(self) -> "DeviceMesh":
        if self._device_mesh is None:
            raise RuntimeError("Accessing the device mesh before processes have initialized is not allowed.")
        return self._device_mesh

    @property
    @override
    def checkpoint_io(self) -> CheckpointIO:
        raise NotImplementedError(f"The `{type(self).__name__}` does not use the `CheckpointIO` plugin interface.")

    @checkpoint_io.setter
    @override
    def checkpoint_io(self, io: CheckpointIO) -> None:
        raise NotImplementedError(f"The `{type(self).__name__}` does not support setting a `CheckpointIO` plugin.")

    @property
    @override
    def root_device(self) -> torch.device:
        assert self.parallel_devices is not None
        return self.parallel_devices[self.local_rank]

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Move any device_mesh access to inside the launched function, after fabric.setup / setup_environment has run
2. If you need a custom mesh, construct and pass your own DeviceMesh to the strategy rather than reading it early
3. Gate mesh-dependent code on mesh availability (check strategy._device_mesh or catch the error) during early phases

Example fix

# before
strategy = ModelParallelStrategy()
mesh = strategy.device_mesh  # RuntimeError: not initialized yet
fabric = Fabric(strategy=strategy)

# after
strategy = ModelParallelStrategy()
fabric = Fabric(strategy=strategy)
with fabric.init_module():
    model = MyModel()
# mesh is available after setup:
mesh = strategy.device_mesh

if strategy._device_mesh is None:  # not yet set up
    fabric.setup_environment()  # or wait until fabric.setup()/run()

def mesh_ready(strategy) -> bool:
    return getattr(strategy, "_device_mesh", None) is not None

try:
    mesh = strategy.device_mesh
except RuntimeError:
    mesh = None  # defer mesh-dependent work until after setup

When it happens

Trigger: Accessing strategy.device_mesh in __init__, at config time, or before calling fabric.setup()/run() (i.e., before setup_environment executes) on ModelParallelStrategy.

Common situations: Trying to inspect or pass the mesh to other components right after constructing the strategy; logging/mesh introspection utilities that run pre-setup; using tensor-parallel APIs that expect a mesh before Lightning created it.

Related errors

• Accessing the device mesh before processes have initialized
• The `{type(self).__name__}` does not use the `CheckpointIO`
• The sizes `data_parallel_size={data_parallel_size}` and `ten
• The `ModelParallelStrategy` does not support `Fabric(..., pr
• Device should be CPU, got {device} instead.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d1f8ac54b55608c8. Report an issue: GitHub.