facebookresearch/detectron2 · error · OSError

File {path} not found on main worker.

Error message

File {path} not found on main worker.

What it means

In distributed training, the main (rank 0) worker verifies the checkpoint path exists locally after path resolution. If rank 0 cannot see the file, loading aborts with this OSError so workers don't deadlock waiting for a checkpoint only some ranks possess.

Source

Thrown at detectron2/checkpoint/detection_checkpoint.py:45

            save_dir,
            save_to_disk=is_main_process if save_to_disk is None else save_to_disk,
            **checkpointables,
        )
        self.path_manager = PathManager
        self._parsed_url_during_load = None

    def load(self, path, *args, **kwargs):
        assert self._parsed_url_during_load is None
        need_sync = False
        logger = logging.getLogger(__name__)
        logger.info("[DetectionCheckpointer] Loading from {} ...".format(path))

        if path and isinstance(self.model, DistributedDataParallel):
            path = self.path_manager.get_local_path(path)
            has_file = os.path.isfile(path)
            all_has_file = comm.all_gather(has_file)
            if not all_has_file[0]:
                raise OSError(f"File {path} not found on main worker.")
            if not all(all_has_file):
                logger.warning(
                    f"Not all workers can read checkpoint {path}. "
                    "Training may fail to fully resume."
                )
                # TODO: broadcast the checkpoint file contents from main
                # worker, and load from it instead.
                need_sync = True
            if not has_file:
                path = None  # don't load if not readable

        if path:
            parsed_url = urlparse(path)
            self._parsed_url_during_load = parsed_url
            path = parsed_url._replace(query="").geturl()  # remove query from filename
            path = self.path_manager.get_local_path(path)
        ret = super().load(path, *args, **kwargs)

View on GitHub (pinned to a2f4a8771a)

Solutions

1. Verify the path exists on the main worker node: ls <path> on rank 0's machine
2. Use an absolute path on a filesystem mounted on all nodes, or a URL that PathManager can download
3. If the checkpoint is gone, restart from a different checkpoint or from scratch instead of resuming

Example fix

# before
trainer.resume_or_load(resume=True)  # LAST checkpoint path stale on rank 0
# after
# ensure the file is visible, then pass explicit path
trainer.resume_or_load(resume=False)
trainer.checkpointer.load("/shared/ckpt/model_0099.pth")

import os, torch.distributed as dist
local = path if os.path.isfile(path) else None
if local is None:
    raise FileNotFoundError(f"checkpoint missing on rank {dist.get_rank()}: {path}")
# optionally all-reduce file existence before calling load
has = torch.tensor(int(os.path.isfile(path)), device='cuda')
torch.distributed.all_reduce(has, op=torch.distributed.ReduceOp.MIN)
assert has.item() == 1

try:
    trainer.resume_or_load(resume=True)
except OSError as e:
    if "not found on main worker" in str(e):
        trainer.resume_or_load(resume=False)  # start fresh
    else:
        raise

When it happens

Trigger: Calling DetectionCheckpointer.load(path) where model is wrapped in DistributedDataParallel and os.path.isfile(path) is False on rank 0 — e.g. wrong path, NFS not mounted on the main node, or a URL whose local cache failed to download on rank 0.

Common situations: Resuming training with a mistyped checkpoint path; shared filesystem mount missing on the head node; S3/GS download failure only on rank 0; relative paths resolved differently per node.

Related errors

• Cannot match one checkpoint key to multiple keys in the mode
• Unsupported query remaining: f{queries}, orginal filename: {
• Cannot import name {relative_import_path} from {original_fil

AI-assisted analysis of facebookresearch/detectron2@a2f4a8771a (2026-08-27). Data as JSON: /api/errors/86c6fe9290324fe3. Report an issue: GitHub.