{"record":{"id":"457bd453d816ca8e","repo":"Lightning-AI/pytorch-lightning","slug":"multihost-setups-do-not-have-a-shared-filesystem","errorCode":null,"errorMessage":"Multihost setups do not have a shared filesystem, so the checkpoint shards cannot be consolidated into a single checkpoint after saving them. Please switch to `XLAFSDPStrategy(state_dict_type='sharded')`. TIP: You can consolidate them manually by getting them together into a single directory and running `python -m torch_xla.distributed.fsdp.consolidate_sharded_ckpts --ckpt_prefix {ckpt_prefix!r} --ckpt_suffix {ckpt_suffix!r} --save_path 'path/to/consolidated.ckpt'`.","messagePattern":"Multihost setups do not have a shared filesystem, so the checkpoint shards cannot be consolidated into a single checkpoint after saving them\\. Please switch to `XLAFSDPStrategy\\(state_dict_type='sharded'\\)`\\. TIP: You can consolidate them manually by getting them together into a single directory and running `python -m torch_xla\\.distributed\\.fsdp\\.consolidate_sharded_ckpts --ckpt_prefix (.+?) --ckpt_suffix (.+?) --save_path 'path/to/consolidated\\.ckpt'`\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/strategies/xla_fsdp.py","lineNumber":463,"sourceCode":"        # ensure model parameters are updated\n        xm.mark_step()\n\n        parallel_devices = self.parallel_devices\n        assert parallel_devices is not None\n        if self._sequential_save:\n            # each host runs this in parallel, but the ranks in the host run it sequentially\n            for rank in range(len(parallel_devices)):\n                if rank == self.local_rank:\n                    self._save_checkpoint_shard(path, state, storage_options, filter)\n                self.barrier(f\"wait-for-{rank}-save\")\n        else:\n            self._save_checkpoint_shard(path, state, storage_options, filter)\n\n        if self._state_dict_type == \"full\":\n            ckpt_prefix = str(path / \"checkpoint\")\n            ckpt_suffix = \"_rank-*-of-*.pth\"\n            if len(parallel_devices) != self.world_size:  # multihost\n                raise OSError(\n                    \"Multihost setups do not have a shared filesystem, so the checkpoint shards cannot be consolidated\"\n                    \" into a single checkpoint after saving them. Please switch to\"\n                    \" `XLAFSDPStrategy(state_dict_type='sharded')`. TIP: You can consolidate them manually by getting\"\n                    \" them together into a single directory and running `python -m\"\n                    f\" torch_xla.distributed.fsdp.consolidate_sharded_ckpts --ckpt_prefix {ckpt_prefix!r} --ckpt_suffix\"\n                    f\" {ckpt_suffix!r} --save_path 'path/to/consolidated.ckpt'`.\"\n                )\n\n            from torch_xla.distributed.fsdp import consolidate_sharded_model_checkpoints\n\n            self.barrier(\"before_ckpt_consolidation\")\n            if self.is_global_zero:\n                save_path = path.parent / \"consolidated.ckpt\"\n                # save consolidated checkpoint separate to the shards\n                consolidate_sharded_model_checkpoints(ckpt_prefix, ckpt_suffix, str(save_path))\n                # remove the shards directory\n                self.checkpoint_io.remove_checkpoint(path)\n                # mv the consolidated checkpoint where the user would expect it","sourceCodeStart":445,"sourceCodeEnd":481,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/strategies/xla_fsdp.py#L445-L481","documentation":"Raised when saving a 'full' (consolidated) XLA FSDP checkpoint on a multihost TPU setup where len(parallel_devices) != world_size. Full checkpoints require gathering and consolidating shards on one host, which is impossible without a shared filesystem across hosts. The error suggests switching to sharded state dicts and consolidating manually afterwards.","triggerScenarios":"Using XLAFSDPStrategy(state_dict_type='full') (the default) with XLAFSDPStrategy.run(save) on a multihost TPU environment (e.g. TPU Pod slices via torchrun/tpu_pod_launch) where each host only sees its local ranks.","commonSituations":"Scaling single-host TPU training to a TPU Pod; CI or scripts that worked on v8-8 failing on v4-32/pod slices; default state_dict_type left as 'full' when moving to multihost.","solutions":["Construct the strategy with sharded state dicts: XLAFSDPStrategy(state_dict_type='sharded'), then save per-rank shards","Alternatively save the shards, gather them into one directory, and run: python -m torch_xla.distributed.fsdp.consolidate_sharded_ckpts --ckpt_prefix <dir>/checkpoint --ckpt_suffix '_rank-*-of-*.pth' --save_path consolidated.ckpt","If you truly need full checkpoints on one host, restrict training to single-host setups (v8 TPU with all ranks visible to one process)"],"exampleFix":"# before\nstrategy = XLAFSDPStrategy(state_dict_type='full')\n\n# after\nstrategy = XLAFSDPStrategy(state_dict_type='sharded')","handlingStrategy":"validation","validationCode":"strategy = XLAFSDPStrategy(state_dict_type='sharded' if len(devices) != world_size else 'full')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Configure state_dict_type='sharded' for any multihost TPU run","Automate post-hoc consolidation in your training script using torch_xla.distributed.fsdp.consolidate_sharded_ckpts","Record world size in checkpoint metadata to catch mismatches on resume"],"tags":["xla","fsdp","tpu","multihost","checkpoint","consolidation","lightning-fabric"],"backgroundTag":"distributed-checkpoint-consolidation-unsupported","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}