Lightning-AI/pytorch-lightning · error · ValueError

The path {str(path)!r} does not point to a valid checkpoint.

Error message

The path {str(path)!r} does not point to a valid checkpoint. Make sure the path points to either a directory with distributed checkpoint shards, or a single file with a full checkpoint.

What it means

After attempting both distributed-sharded and single-file loading paths, _load_checkpoint found neither a directory of dist-Checkpoint shards nor a loadable full checkpoint file at `path`. The final fallback raises this ValueError indicating the path is not a valid checkpoint location.

Source

Thrown at src/lightning/fabric/strategies/model_parallel.py:511

            optimizer_state = _rekey_optimizer_state_if_needed(optimizer_state, module)
            set_optimizer_state_dict(
                module,
                optimizer,
                optim_state_dict=optimizer_state,
                options=state_dict_options,
            )

        requested_metadata_keys = state.keys() - modules.keys() - optimizers.keys()
        _validate_keys_for_strict_loading(requested_metadata_keys, checkpoint.keys(), strict=strict)

        # Load metadata (anything not a module or optimizer)
        _move_state_into(source=checkpoint, destination=state, keys=requested_metadata_keys)

        # return the remaining metadata that wasn't requested as part of `state`
        return checkpoint

    raise ValueError(
        f"The path {str(path)!r} does not point to a valid checkpoint. Make sure the path points to either a"
        " directory with distributed checkpoint shards, or a single file with a full checkpoint."
    )


def _setup_device_mesh(
    data_parallel_size: int,
    tensor_parallel_size: int,
    world_size: int,
    device: torch.device,
) -> "DeviceMesh":
    from torch.distributed.device_mesh import init_device_mesh

    if data_parallel_size * tensor_parallel_size != world_size:
        raise RuntimeError(
            f"The sizes `data_parallel_size={data_parallel_size}` and"
            f" `tensor_parallel_size={tensor_parallel_size}` multiplied should equal the world size"
            f" ({world_size})."

View on GitHub (pinned to 9fed5c27d2)

Solutions

Verify the path exists and inspect its contents (directory with .distcp shards or a single full file)
If the checkpoint was moved, re-save or update the path; for remote paths check the fsspec filesystem permissions/existence
Re-create the checkpoint with strategy.save_checkpoint under the same ModelParallel strategy
Fall back to torch.load on a regular checkpoint if the run doesn't need distributed loading

Example fix

# before
strategy.load_checkpoint('checkpoints/last', state={'model': model})
# after
from pathlib import Path
assert Path('checkpoints/last').exists() and any(Path('checkpoints/last').iterdir()), 'bad checkpoint dir'
strategy.load_checkpoint('checkpoints/last', state={'model': model})

Defensive patterns

Strategy: validation

Validate before calling

from pathlib import Path

p = Path(path)
assert p.exists(), f'checkpoint path missing: {p}'
if p.is_dir():
    assert any(p.iterdir()), 'shard directory is empty'
else:
    assert p.is_file() and p.stat().st_size > 0, 'not a valid checkpoint file'

Try / catch

try:
    strategy.load_checkpoint(path, state={'model': model})
except ValueError as e:
    if 'does not point to a valid checkpoint' in str(e):
        raise FileNotFoundError(path) from e
    raise

Prevention

Validate checkpoint paths (including fsspec URLs) before resume logic
Fail fast on missing checkpoints at startup rather than mid-training

When it happens

Trigger: load_checkpoint called with a path that doesn't exist, is an empty directory, contains neither __0_0.distcp-style shards nor a full checkpoint file, or points to a remote URL whose contents aren't a valid checkpoint.

Common situations: Typo'd or stale checkpoint path; resuming a run whose checkpoint files were deleted/moved; pointing at an fsspec/S3 URL where the files are missing; passing a checkpoint saved by a different (non-distributed) mechanism.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/cdf61554b43f8470. Report an issue: GitHub.