{"record":{"id":"9a7d12001181b057","repo":"Lightning-AI/pytorch-lightning","slug":"accessing-the-device-mesh-before-processes-have-in-9a7d12","errorCode":null,"errorMessage":"Accessing the device mesh before processes have initialized is not allowed.","messagePattern":"Accessing the device mesh before processes have initialized is not allowed\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/lightning/pytorch/strategies/model_parallel.py","lineNumber":106,"sourceCode":"        data_parallel_size: Union[Literal[\"auto\"], int] = \"auto\",\n        tensor_parallel_size: Union[Literal[\"auto\"], int] = \"auto\",\n        save_distributed_checkpoint: bool = True,\n        process_group_backend: Optional[str] = None,\n        timeout: Optional[timedelta] = default_pg_timeout,\n    ) -> None:\n        super().__init__()\n        self._data_parallel_size = data_parallel_size\n        self._tensor_parallel_size = tensor_parallel_size\n        self._save_distributed_checkpoint = save_distributed_checkpoint\n        self._process_group_backend: Optional[str] = process_group_backend\n        self._timeout: Optional[timedelta] = timeout\n        self._device_mesh: Optional[DeviceMesh] = None\n        self.num_nodes = 1\n\n    @property\n    def device_mesh(self) -> \"DeviceMesh\":\n        if self._device_mesh is None:\n            raise RuntimeError(\"Accessing the device mesh before processes have initialized is not allowed.\")\n        return self._device_mesh\n\n    @property\n    @override\n    def root_device(self) -> torch.device:\n        assert self.parallel_devices is not None\n        return self.parallel_devices[self.local_rank]\n\n    @property\n    def num_processes(self) -> int:\n        return len(self.parallel_devices) if self.parallel_devices is not None else 0\n\n    @property\n    @override\n    def distributed_sampler_kwargs(self) -> dict[str, Any]:\n        assert self.device_mesh is not None\n        data_parallel_mesh = self.device_mesh[\"data_parallel\"]\n        return {\"num_replicas\": data_parallel_mesh.size(), \"rank\": data_parallel_mesh.get_local_rank()}","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/pytorch/strategies/model_parallel.py#L88-L124","documentation":"ModelParallelStrategy exposes a device_mesh property that returns the torch.distributed.device_mesh.DeviceMesh created during setup. Accessing it before the strategy has initialized (mesh not yet built) raises RuntimeError, because the mesh depends on process group initialization that only happens at setup time.","triggerScenarios":"Reading strategy.device_mesh in LightningModule.__init__, configure_model (before setup), or any code executed before trainer setup (e.g. in module-level code or a pre-run callback) with ModelParallelStrategy.","commonSituations":"Trying to inspect or pass the mesh to torch tensor creation in __init__ or configure_model; logging mesh info before training starts.","solutions":["Move mesh access into/after setup: use setup() hooks, on_fit_start callbacks, or access it inside forward/training_step where setup already ran","Create your own DeviceMesh and pass parallelize_to_plan / use the strategy's API for pre-setup mesh needs","Guard with an attribute check: getattr(strategy, '_device_mesh', None) before use"],"exampleFix":"# before\nclass LitModel(L.LightningModule):\n    def __init__(self):\n        mesh = self.trainer.strategy.device_mesh  # RuntimeError\n\n# after\nclass LitModel(L.LightningModule):\n    def setup(self, stage=None):\n        mesh = self.trainer.strategy.device_mesh  # OK: after strategy setup\n","handlingStrategy":"validation","validationCode":"mesh = getattr(strategy, \"_device_mesh\", None)\nif mesh is None:\n    # too early: defer access to setup()/hooks\n    ...","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Only read strategy.device_mesh from setup() or run-phase hooks (on_fit_start, training_step)","Create an explicit DeviceMesh yourself if you need one pre-setup"],"tags":["device-mesh","model-parallel","lifecycle","pytorch-lightning"],"backgroundTag":"access-before-initialization","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}