{"record":{"id":"86c6fe9290324fe3","repo":"facebookresearch/detectron2","slug":"file-path-not-found-on-main-worker","errorCode":null,"errorMessage":"File {path} not found on main worker.","messagePattern":"File (.+?) not found on main worker\\.","errorType":"exception","errorClass":"OSError","httpStatus":null,"severity":"error","filePath":"detectron2/checkpoint/detection_checkpoint.py","lineNumber":45,"sourceCode":"            save_dir,\n            save_to_disk=is_main_process if save_to_disk is None else save_to_disk,\n            **checkpointables,\n        )\n        self.path_manager = PathManager\n        self._parsed_url_during_load = None\n\n    def load(self, path, *args, **kwargs):\n        assert self._parsed_url_during_load is None\n        need_sync = False\n        logger = logging.getLogger(__name__)\n        logger.info(\"[DetectionCheckpointer] Loading from {} ...\".format(path))\n\n        if path and isinstance(self.model, DistributedDataParallel):\n            path = self.path_manager.get_local_path(path)\n            has_file = os.path.isfile(path)\n            all_has_file = comm.all_gather(has_file)\n            if not all_has_file[0]:\n                raise OSError(f\"File {path} not found on main worker.\")\n            if not all(all_has_file):\n                logger.warning(\n                    f\"Not all workers can read checkpoint {path}. \"\n                    \"Training may fail to fully resume.\"\n                )\n                # TODO: broadcast the checkpoint file contents from main\n                # worker, and load from it instead.\n                need_sync = True\n            if not has_file:\n                path = None  # don't load if not readable\n\n        if path:\n            parsed_url = urlparse(path)\n            self._parsed_url_during_load = parsed_url\n            path = parsed_url._replace(query=\"\").geturl()  # remove query from filename\n            path = self.path_manager.get_local_path(path)\n        ret = super().load(path, *args, **kwargs)\n","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/facebookresearch/detectron2/blob/a2f4a8771ab77e8411c26b27f24f9489a28a2453/detectron2/checkpoint/detection_checkpoint.py#L27-L63","documentation":"In distributed training, the main (rank 0) worker verifies the checkpoint path exists locally after path resolution. If rank 0 cannot see the file, loading aborts with this OSError so workers don't deadlock waiting for a checkpoint only some ranks possess.","triggerScenarios":"Calling DetectionCheckpointer.load(path) where model is wrapped in DistributedDataParallel and os.path.isfile(path) is False on rank 0 — e.g. wrong path, NFS not mounted on the main node, or a URL whose local cache failed to download on rank 0.","commonSituations":"Resuming training with a mistyped checkpoint path; shared filesystem mount missing on the head node; S3/GS download failure only on rank 0; relative paths resolved differently per node.","solutions":["Verify the path exists on the main worker node: ls <path> on rank 0's machine","Use an absolute path on a filesystem mounted on all nodes, or a URL that PathManager can download","If the checkpoint is gone, restart from a different checkpoint or from scratch instead of resuming"],"exampleFix":"# before\ntrainer.resume_or_load(resume=True)  # LAST checkpoint path stale on rank 0\n# after\n# ensure the file is visible, then pass explicit path\ntrainer.resume_or_load(resume=False)\ntrainer.checkpointer.load(\"/shared/ckpt/model_0099.pth\")","handlingStrategy":"validation","validationCode":"import os, torch.distributed as dist\nlocal = path if os.path.isfile(path) else None\nif local is None:\n    raise FileNotFoundError(f\"checkpoint missing on rank {dist.get_rank()}: {path}\")\n# optionally all-reduce file existence before calling load\nhas = torch.tensor(int(os.path.isfile(path)), device='cuda')\ntorch.distributed.all_reduce(has, op=torch.distributed.ReduceOp.MIN)\nassert has.item() == 1","typeGuard":null,"tryCatchPattern":"try:\n    trainer.resume_or_load(resume=True)\nexcept OSError as e:\n    if \"not found on main worker\" in str(e):\n        trainer.resume_or_load(resume=False)  # start fresh\n    else:\n        raise","preventionTips":["Use absolute paths on shared storage mounted on all nodes","Verify checkpoint existence on every node before launching distributed jobs","Monitor download success of remote checkpoints (S3/GS) before training starts"],"tags":["distributed","checkpoint","file-not-found","ddp"],"backgroundTag":"checkpoint-file-missing-on-worker","analyzedSha":"a2f4a8771ab77e8411c26b27f24f9489a28a2453","analyzedAt":"2026-08-27T12:08:21.260Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}