{"record":{"id":"d7a38fe69f5c066d","repo":"Lightning-AI/pytorch-lightning","slug":"upgrade-fsspec-to-enable-cross-device-local-checkp","errorCode":null,"errorMessage":"Upgrade fsspec to enable cross-device local checkpoints: pip install \"fsspec[http]>=2025.5.0\"","messagePattern":"Upgrade fsspec to enable cross-device local checkpoints: pip install \"fsspec\\[http\\]>=2025\\.5\\.0\"","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/utilities/cloud_io.py","lineNumber":130,"sourceCode":"                bytesbuffer = io.BytesIO()\n                torch.save(checkpoint, bytesbuffer)\n                if is_azure:\n                    # Azure uses a plain write because adlfs stages blocks sequentially, making\n                    # pipe() slower.\n                    with fs.open(urlpath, \"wb\") as f:\n                        f.write(bytesbuffer.getvalue())\n                else:\n                    # Use fs.pipe() for S3/GCS where it triggers parallel multipart uploads,\n                    # giving 4-5x throughput improvement for checkpoints >= 500 MB.\n                    fs.pipe(urlpath, bytesbuffer.getvalue())\n            else:\n                # Stream directly to the file so we never hold a second full copy of the checkpoint\n                # in memory. This matters for large FSDP/ModelParallel full state dicts on local disk.\n                with fs.open(urlpath, \"wb\") as f:\n                    torch.save(checkpoint, f)\n    except PermissionError as e:\n        if isinstance(e.__context__, OSError) and getattr(e.__context__, \"errno\", None) == errno.EXDEV:\n            raise RuntimeError(\n                'Upgrade fsspec to enable cross-device local checkpoints: pip install \"fsspec[http]>=2025.5.0\"',\n            ) from e\n\n\ndef _is_object_storage(fs: AbstractFileSystem) -> bool:\n    if module_available(\"adlfs\"):\n        from adlfs import AzureBlobFileSystem\n\n        if isinstance(fs, AzureBlobFileSystem):\n            return True\n\n    if module_available(\"gcsfs\"):\n        from gcsfs import GCSFileSystem\n\n        if isinstance(fs, GCSFileSystem):\n            return True\n\n    if module_available(\"s3fs\"):","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/utilities/cloud_io.py#L112-L148","documentation":"Raised by _atomic_save when torch.save to an fsspec URL fails with PermissionError whose underlying cause is OSError errno EXDEV (cross-device link). Older fsspec versions implement local 'atomic' saves via rename across a temp dir, which fails across filesystems; fsspec>=2025.5.0 can stream via its HTTP/local pipe machinery instead. Upgrading fsspec is the documented fix.","triggerScenarios":"Saving a checkpoint to a local path on a different filesystem/mount than the temp directory (e.g. path under a network mount, docker volume, or /dev/shm-adjacent mount) with fsspec < 2025.5.0 installed — commonly triggered from fabric.save_checkpoint / strategy save flows and DCP consolidation (_consolidate).","commonSituations":"Kubernetes/docker deployments writing checkpoints to mounted volumes; CI writing to workspace mounts; environments where TMPDIR and the checkpoint dir live on different devices; recent Lightning versions routing local saves through fsspec.","solutions":["pip install \"fsspec[http]>=2025.5.0\" (the message's recommended fix)","Alternatively set TMPDIR (or pass a temp dir on the same filesystem) so the atomic rename stays on one device","Or save to a plain local path on the same filesystem as the temp dir and copy afterwards"],"exampleFix":"# before\n$ pip install fsspec==2023.12.2  # atomic save raises EXDEV on cross-device mounts\n\n# after\n$ pip install \"fsspec[http]>=2025.5.0\"","handlingStrategy":"fallback","validationCode":"import fsspec\nfrom packaging.version import Version\nassert Version(fsspec.__version__) >= Version('2025.5.0'), 'upgrade fsspec for cross-device atomic saves'","typeGuard":null,"tryCatchPattern":"try:\n    fabric.save_checkpoint(path, state)\nexcept RuntimeError as e:\n    if 'fsspec' in str(e):\n        fabric.save_checkpoint(local_path, state)  # save locally, move later\n    else:\n        raise","preventionTips":["Pin fsspec[http]>=2025.5.0 in requirements","Set TMPDIR to the checkpoint filesystem in containerized jobs","Smoke-test checkpoint saving in new deployment environments"],"tags":["fsspec","checkpoint","cross-device","exdev","atomic-save","environment","lightning-fabric"],"backgroundTag":"cross-device-link-exdev","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}