Lightning-AI/pytorch-lightning · error · RuntimeError

Upgrade fsspec to enable cross-device local checkpoints: pip

Error message

Upgrade fsspec to enable cross-device local checkpoints: pip install "fsspec[http]>=2025.5.0"

What it means

Raised by _atomic_save when torch.save to an fsspec URL fails with PermissionError whose underlying cause is OSError errno EXDEV (cross-device link). Older fsspec versions implement local 'atomic' saves via rename across a temp dir, which fails across filesystems; fsspec>=2025.5.0 can stream via its HTTP/local pipe machinery instead. Upgrading fsspec is the documented fix.

Source

Thrown at src/lightning/fabric/utilities/cloud_io.py:130

                bytesbuffer = io.BytesIO()
                torch.save(checkpoint, bytesbuffer)
                if is_azure:
                    # Azure uses a plain write because adlfs stages blocks sequentially, making
                    # pipe() slower.
                    with fs.open(urlpath, "wb") as f:
                        f.write(bytesbuffer.getvalue())
                else:
                    # Use fs.pipe() for S3/GCS where it triggers parallel multipart uploads,
                    # giving 4-5x throughput improvement for checkpoints >= 500 MB.
                    fs.pipe(urlpath, bytesbuffer.getvalue())
            else:
                # Stream directly to the file so we never hold a second full copy of the checkpoint
                # in memory. This matters for large FSDP/ModelParallel full state dicts on local disk.
                with fs.open(urlpath, "wb") as f:
                    torch.save(checkpoint, f)
    except PermissionError as e:
        if isinstance(e.__context__, OSError) and getattr(e.__context__, "errno", None) == errno.EXDEV:
            raise RuntimeError(
                'Upgrade fsspec to enable cross-device local checkpoints: pip install "fsspec[http]>=2025.5.0"',
            ) from e


def _is_object_storage(fs: AbstractFileSystem) -> bool:
    if module_available("adlfs"):
        from adlfs import AzureBlobFileSystem

        if isinstance(fs, AzureBlobFileSystem):
            return True

    if module_available("gcsfs"):
        from gcsfs import GCSFileSystem

        if isinstance(fs, GCSFileSystem):
            return True

    if module_available("s3fs"):

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. pip install "fsspec[http]>=2025.5.0" (the message's recommended fix)
2. Alternatively set TMPDIR (or pass a temp dir on the same filesystem) so the atomic rename stays on one device
3. Or save to a plain local path on the same filesystem as the temp dir and copy afterwards

Example fix

# before
$ pip install fsspec==2023.12.2  # atomic save raises EXDEV on cross-device mounts

# after
$ pip install "fsspec[http]>=2025.5.0"

import fsspec
from packaging.version import Version
assert Version(fsspec.__version__) >= Version('2025.5.0'), 'upgrade fsspec for cross-device atomic saves'

try:
    fabric.save_checkpoint(path, state)
except RuntimeError as e:
    if 'fsspec' in str(e):
        fabric.save_checkpoint(local_path, state)  # save locally, move later
    else:
        raise

When it happens

Trigger: Saving a checkpoint to a local path on a different filesystem/mount than the temp directory (e.g. path under a network mount, docker volume, or /dev/shm-adjacent mount) with fsspec < 2025.5.0 installed — commonly triggered from fabric.save_checkpoint / strategy save flows and DCP consolidation (_consolidate).

Common situations: Kubernetes/docker deployments writing checkpoints to mounted volumes; CI writing to workspace mounts; environments where TMPDIR and the checkpoint dir live on different devices; recent Lightning versions routing local saves through fsspec.

Related errors

• Could not find a XLAFSDP model in the provided checkpoint st
• Found multiple XLAFSDP modules in the given state. Saving ch
• Multihost setups do not have a shared filesystem, so the che
• Got `XLAFSDPStrategy.load_checkpoint(..., state={state!r})`
• Loading a single module or optimizer object from a checkpoin

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d7a38fe69f5c066d. Report an issue: GitHub.