Lightning-AI/pytorch-lightning · error · ImportError

Remote (fsspec) distributed checkpoints require `torch.distr

Error message

Remote (fsspec) distributed checkpoints require `torch.distributed.checkpoint._fsspec_filesystem`, which is not available in this PyTorch build. Use a local checkpoint path or upgrade PyTorch.

What it means

Raised by _import_fsspec_dcp_filesystem when importing torch.distributed.checkpoint._fsspec_filesystem fails; this private module is needed to write/read torch.distributed.checkpoint (DCP) shards directly to remote (fsspec) filesystems like s3:// or gs://. Builds/versions of PyTorch lacking it cannot do remote DCP I/O, so Lightning converts the deep ImportError into an actionable message: use a local path or upgrade PyTorch.

Source

Thrown at src/lightning/fabric/utilities/cloud_io.py:282

    if _is_local_file_protocol(str(path)):
        from torch.distributed.checkpoint import FileSystemReader

        return FileSystemReader(path=path)
    FsspecReader = _import_fsspec_dcp_filesystem("FsspecReader")
    return FsspecReader(path=str(path))


def _import_fsspec_dcp_filesystem(name: str) -> Any:
    """Import ``FsspecReader``/``FsspecWriter`` from torch's private DCP fsspec module.

    These live in a private module that not every PyTorch build ships, so raise an actionable error
    instead of letting a bare ``ImportError`` surface from deep in the call stack.

    """
    try:
        module = importlib.import_module("torch.distributed.checkpoint._fsspec_filesystem")
    except ImportError as e:
        raise ImportError(
            "Remote (fsspec) distributed checkpoints require"
            " `torch.distributed.checkpoint._fsspec_filesystem`, which is not available in this"
            " PyTorch build. Use a local checkpoint path or upgrade PyTorch."
        ) from e
    return getattr(module, name)

View on GitHub (pinned to 9fed5c27d2)

Solutions

Save distributed checkpoints to a local path first, then upload (aws s3 cp / gsutil cp) to remote storage
Upgrade PyTorch to a version that includes torch.distributed.checkpoint._fsspec_filesystem support for remote filesystems
Verify with: python -c "import torch.distributed.checkpoint._fsspec_filesystem" before attempting remote DCP saves

Example fix

# before
fabric.save_checkpoint('s3://bucket/run/ckpt', state=state)  # ImportError on old torch

# after
fabric.save_checkpoint('/tmp/run/ckpt', state=state)
subprocess.run(['aws', 's3', 'cp', '--recursive', '/tmp/run/ckpt', 's3://bucket/run/ckpt'])

Defensive patterns

Strategy: fallback

Validate before calling

import importlib.util
remote_ok = importlib.util.find_spec('torch.distributed.checkpoint._fsspec_filesystem') is not None
if not remote_ok:
    ckpt_path = local_staging_dir  # save locally, then upload

Try / catch

try:
    fabric.save_checkpoint('s3://bucket/ckpt', state=state)
except ImportError:
    fabric.save_checkpoint('/tmp/ckpt', state=state)
    upload_to_s3('/tmp/ckpt')

Prevention

Check for the private DCP fsspec module before adopting remote DCP paths
Keep a local-staging + upload fallback in checkpoint utilities

When it happens

Trigger: Calling save_checkpoint/load_checkpoint with a remote URL (s3://bucket/..., gs://...) on a distributed checkpoint path while the installed PyTorch does not expose torch.distributed.checkpoint._fsspec_filesystem (older PyTorch versions, some nightly/modified builds).

Common situations: Migrating distributed checkpoints from local disk to S3/GCS with an older PyTorch; environment pinning PyTorch < 2.x/older 2.x; slim wheels or forks where the private module was removed/renamed.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/73a3537bbc8fc450. Report an issue: GitHub.