Lightning-AI/pytorch-lightning · error · ValueError

The path {str(file)!r} does not point to valid sharded check

Error message

The path {str(file)!r} does not point to valid sharded checkpoints. Make sure the path points to a directory with XLAFSDP checkpoint shards.

What it means

When XLAFSDPStrategy is configured with state_dict_type='sharded', each rank loads its own shard file named checkpoint_rank-{global_rank:08d}-of-{world_size:08d}.pth inside the checkpoint directory. If that per-rank file does not exist at the expected path, loading cannot proceed. Common causes include pointing at the wrong directory, a different world size than when saving, or only full/consolidated checkpoints being present.

Source

Thrown at src/lightning/fabric/strategies/xla_fsdp.py:550

            )

        # broadcast the path from rank 0 to ensure all the states are loaded from a common path
        path = Path(self.broadcast(path))

        if isinstance(state, (Module, Optimizer)):
            raise NotImplementedError(
                "Loading a single module or optimizer object from a checkpoint"
                " is not supported yet with the XLAFSDP strategy."
            )

        from torch_xla.distributed.fsdp import XlaFullyShardedDataParallel as XLAFSDP

        modules = {key: module for key, module in state.items() if isinstance(module, XLAFSDP)}
        optimizers = {key: optim for key, optim in state.items() if isinstance(optim, Optimizer)}
        if self._state_dict_type == "sharded":
            file = path / f"checkpoint_rank-{self.global_rank:08d}-of-{self.world_size:08d}.pth"
            if not file.is_file():
                raise ValueError(
                    f"The path {str(file)!r} does not point to valid sharded checkpoints. Make sure the path points to"
                    " a directory with XLAFSDP checkpoint shards."
                )
            if len(modules) == 0:
                raise ValueError(
                    "Could not find a XLAFSDP model in the provided checkpoint state. Please provide the model as"
                    " part of the state like so: `load_checkpoint(..., state={'model': model, ...})`. Make sure"
                    " you set up the model (and optimizers if any) through the strategy before loading the checkpoint."
                )
            if len(modules) > 1:
                raise ValueError(
                    "Found multiple XLAFSDP modules in the given state. Loading checkpoints with FSDP is"
                    " currently limited to a single model per checkpoint. To load multiple models, call the"
                    " load method for each model separately with a different path."
                )

            _, module = list(modules.items())[0]
            sharded_ckpt = torch.load(file)

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Verify the path is the directory containing checkpoint_rank-XXXXXXXX-of-XXXXXXXX.pth files from the save (check filenames' rank-of-world_size parts match the current run)
2. Match the world size (and state_dict_type) used at save time, or re-shard/re-save the checkpoint accordingly
3. If you only have a consolidated full checkpoint, load with state_dict_type='full' instead, or re-generate shards

Example fix

# before
strategy = XLAFSDPStrategy(state_dict_type='sharded')
fabric.load_checkpoint('ckpt/consolidated.ckpt', state={'model': model})

# after
strategy = XLAFSDPStrategy(state_dict_type='full')
fabric.load_checkpoint('ckpt/consolidated.ckpt', state={'model': model})
# or point at the shard directory when using 'sharded'
fabric.load_checkpoint('ckpt/shards/', state={'model': model})

from pathlib import Path
shard = Path(ckpt_dir) / f"checkpoint_rank-{rank:08d}-of-{world_size:08d}.pth"
if not shard.is_file():
    raise FileNotFoundError(f'missing shard {shard}; check world_size and checkpoint dir')

When it happens

Trigger: Calling load_checkpoint on a path that is not the sharded-checkpoint directory, or where the shard for the current rank/world size is missing (e.g. saved with 8 ranks, loading with 4, or passing a consolidated .ckpt file while state_dict_type='sharded').

Common situations: Resuming training after changing TPU/world-size configuration; pointing at the consolidation output file instead of the shards directory; moving checkpoint directories so relative shard files are missing; resuming a 'full' checkpoint with a 'sharded' strategy config (or vice versa).

Related errors

• The path {str(path)!r} does not point to a valid full checkp
• Got `XLAFSDPStrategy.load_checkpoint(..., state={state!r})`
• Loading a single module or optimizer object from a checkpoin
• Could not find a XLAFSDP model in the provided checkpoint st
• Found multiple XLAFSDP modules in the given state. Loading c

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/5785fabe5611e7d0. Report an issue: GitHub.