Lightning-AI/pytorch-lightning · error · FileNotFoundError

Unable to determine if the path belongs to a shared filesyst

Error message

Unable to determine if the path belongs to a shared filesystem. The path does not exist: {path}

What it means

is_shared_filesystem() first broadcasts the path across ranks and requires that the path exists on ALL ranks (verified via reduce_boolean_decision(path.exists(), all=True)). If any rank reports the path missing, it cannot distinguish 'not shared' from 'broken storage', so it raises FileNotFoundError. The path must be created (e.g. by a prior fs.makedirs) before probing.

Source

Thrown at src/lightning/fabric/utilities/distributed.py:72

    """
    # Fast path: Any non-local filesystem is considered shared (e.g., S3)
    if path is not None and not _is_local_file_protocol(path):
        return True

    path = Path(Path.cwd() if path is None else path).resolve()

    # Fast path: Only distributed strategies can detect shared filesystems
    if not hasattr(strategy, "world_size") or strategy.world_size == 1:
        return True

    # Fast path: If the path is not the same on all ranks we know it's not a shared filesystem
    rank_zero_path = strategy.broadcast(path)
    if not strategy.reduce_boolean_decision(rank_zero_path == path, all=True):
        return False

    if not strategy.reduce_boolean_decision(path.exists(), all=True):
        raise FileNotFoundError(
            f"Unable to determine if the path belongs to a shared filesystem. The path does not exist: {path}"
        )

    path = path.parent if path.is_file() else path
    check_file = path / ".lightning_shared_fs_check"
    check_file.unlink(missing_ok=True)

    strategy.barrier()
    if strategy.is_global_zero:
        # Rank 0 creates the file
        check_file.touch()
        found = True
    else:
        # All other ranks will wait until they find the file or timeout
        start = time.perf_counter()
        found = False
        while not found and (time.perf_counter() - start) < timeout:
            found = check_file.exists()

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Create the directory before setup on all ranks: from lightning.fabric.utilities.cloud_io import _fs; _fs.makedirs(path, exist_ok=True) or os.makedirs(path, exist_ok=True) guarded by rank zero + barrier
2. Verify the mount exists on every node (df <path>) before launching
3. Fix typos in the checkpoint/root path

Example fix

# before
fabric = Fabric()
fabric.setup()  # ckpt dir /shared/run1 does not exist everywhere

# after
if fabric.global_rank == 0:
    Path("/shared/run1").mkdir(parents=True, exist_ok=True)
fabric.barrier()
fabric.setup()

from pathlib import Path

ckpt_dir = Path("/shared/run1")
if fabric.global_rank == 0:
    ckpt_dir.mkdir(parents=True, exist_ok=True)
fabric.barrier()
assert ckpt_dir.exists(), f"{ckpt_dir} missing after creation on rank {fabric.global_rank}"

try:
    shared = strategy.is_shared_filesystem(path)
except FileNotFoundError:
    shared = False  # treat as non-shared, fall back to per-rank checkpointing

When it happens

Trigger: Calling strategy.is_shared_filesystem(path) (used by Fabric's checkpoint setup) with a checkpoint directory that hasn't been created yet; a rank on a node where the mount is missing; a typo'd path that exists nowhere.

Common situations: Fabric(...).setup() with a checkpoint path whose parent directory isn't created yet on some node; NFS/Lustre mounts missing on one worker; misconfigured cluster shared storage.

Related errors

• `name` must be a str, found {name}
• `num_nodes` must be a positive integer, but got {num_nodes}.
• Filter should be a dictionary, given {filter!r}
• The filter keys {filter.keys() - state} are not present in t
• Expected `fabric.save(filter=...)` for key {k!r} to be a cal

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/65991c91d966819c. Report an issue: GitHub.