{"record":{"id":"65991c91d966819c","repo":"Lightning-AI/pytorch-lightning","slug":"unable-to-determine-if-the-path-belongs-to-a-share","errorCode":null,"errorMessage":"Unable to determine if the path belongs to a shared filesystem. The path does not exist: {path}","messagePattern":"Unable to determine if the path belongs to a shared filesystem\\. The path does not exist: (.+?)","errorType":"exception","errorClass":"FileNotFoundError","httpStatus":null,"severity":"error","filePath":"src/lightning/fabric/utilities/distributed.py","lineNumber":72,"sourceCode":"\n    \"\"\"\n    # Fast path: Any non-local filesystem is considered shared (e.g., S3)\n    if path is not None and not _is_local_file_protocol(path):\n        return True\n\n    path = Path(Path.cwd() if path is None else path).resolve()\n\n    # Fast path: Only distributed strategies can detect shared filesystems\n    if not hasattr(strategy, \"world_size\") or strategy.world_size == 1:\n        return True\n\n    # Fast path: If the path is not the same on all ranks we know it's not a shared filesystem\n    rank_zero_path = strategy.broadcast(path)\n    if not strategy.reduce_boolean_decision(rank_zero_path == path, all=True):\n        return False\n\n    if not strategy.reduce_boolean_decision(path.exists(), all=True):\n        raise FileNotFoundError(\n            f\"Unable to determine if the path belongs to a shared filesystem. The path does not exist: {path}\"\n        )\n\n    path = path.parent if path.is_file() else path\n    check_file = path / \".lightning_shared_fs_check\"\n    check_file.unlink(missing_ok=True)\n\n    strategy.barrier()\n    if strategy.is_global_zero:\n        # Rank 0 creates the file\n        check_file.touch()\n        found = True\n    else:\n        # All other ranks will wait until they find the file or timeout\n        start = time.perf_counter()\n        found = False\n        while not found and (time.perf_counter() - start) < timeout:\n            found = check_file.exists()","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/Lightning-AI/pytorch-lightning/blob/9fed5c27d2a62ff0efd6c3573599921d6ff67c14/src/lightning/fabric/utilities/distributed.py#L54-L90","documentation":"is_shared_filesystem() first broadcasts the path across ranks and requires that the path exists on ALL ranks (verified via reduce_boolean_decision(path.exists(), all=True)). If any rank reports the path missing, it cannot distinguish 'not shared' from 'broken storage', so it raises FileNotFoundError. The path must be created (e.g. by a prior fs.makedirs) before probing.","triggerScenarios":"Calling strategy.is_shared_filesystem(path) (used by Fabric's checkpoint setup) with a checkpoint directory that hasn't been created yet; a rank on a node where the mount is missing; a typo'd path that exists nowhere.","commonSituations":"Fabric(...).setup() with a checkpoint path whose parent directory isn't created yet on some node; NFS/Lustre mounts missing on one worker; misconfigured cluster shared storage.","solutions":["Create the directory before setup on all ranks: from lightning.fabric.utilities.cloud_io import _fs; _fs.makedirs(path, exist_ok=True) or os.makedirs(path, exist_ok=True) guarded by rank zero + barrier","Verify the mount exists on every node (df <path>) before launching","Fix typos in the checkpoint/root path"],"exampleFix":"# before\nfabric = Fabric()\nfabric.setup()  # ckpt dir /shared/run1 does not exist everywhere\n\n# after\nif fabric.global_rank == 0:\n    Path(\"/shared/run1\").mkdir(parents=True, exist_ok=True)\nfabric.barrier()\nfabric.setup()","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\nckpt_dir = Path(\"/shared/run1\")\nif fabric.global_rank == 0:\n    ckpt_dir.mkdir(parents=True, exist_ok=True)\nfabric.barrier()\nassert ckpt_dir.exists(), f\"{ckpt_dir} missing after creation on rank {fabric.global_rank}\"","typeGuard":null,"tryCatchPattern":"try:\n    shared = strategy.is_shared_filesystem(path)\nexcept FileNotFoundError:\n    shared = False  # treat as non-shared, fall back to per-rank checkpointing","preventionTips":["Create checkpoint dirs on rank 0 followed by a barrier before setup()","Verify mounts on all nodes before launching multi-node jobs"],"tags":["distributed","filesystem","checkpointing","shared-storage"],"backgroundTag":"path-not-found-distributed","analyzedSha":"9fed5c27d2a62ff0efd6c3573599921d6ff67c14","analyzedAt":"2026-08-28T11:52:41.083Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}