{"record":{"id":"dda47abd0a0c6947","repo":"hiyouga/LlamaFactory","slug":"checkpoint-directory-does-not-exist-ckpt-dir","errorCode":null,"errorMessage":"Checkpoint directory does not exist: {ckpt_dir}","messagePattern":"Checkpoint directory does not exist: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/utils/checkpoint.py","lineNumber":301,"sourceCode":"            save_rng_state(ckpt_dir, rank)\n\n        DistributedInterface().sync()\n\n        if rank == 0:\n            mark_checkpoint_complete(ckpt_dir)\n            if self._t.args.save_total_limit is not None:\n                rotate_checkpoints(self._t.args.output_dir, self._t.args.save_total_limit)\n\n        logger.info_rank0(f\"Checkpoint saved to {ckpt_dir}\")\n\n    def resume(self, ckpt_path: str) -> None:\n        \"\"\"Restore full training state from a checkpoint directory.\"\"\"\n        ckpt_dir = resolve_resume_checkpoint_path(ckpt_path, self._t.args.output_dir)\n        if ckpt_dir is None:\n            return\n\n        if not os.path.isdir(ckpt_dir):\n            raise ValueError(f\"Checkpoint directory does not exist: {ckpt_dir}\")\n\n        rank = DistributedInterface().get_rank()\n\n        metadata = load_metadata(ckpt_dir)\n        self._t.global_step = metadata[\"global_step\"]\n        self._t._resume_epoch = metadata[\"epoch\"]\n\n        if self._dist_name in (\"fsdp2\", \"fsdpturbo\", \"deepspeed\"):\n            from ...plugins.trainer_plugins.distributed.interface import DistributedPlugin\n\n            DistributedPlugin(self._dist_name).load_checkpoint(\n                self._t.model,\n                self._t.optimizer,\n                ckpt_dir,\n                processor=self._t.renderer.processor,\n            )\n        else:\n            _load_standard_training_states(","sourceCodeStart":283,"sourceCodeEnd":319,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/utils/checkpoint.py#L283-L319","documentation":"TrainingCheckpointCoordinator.resume() resolves the requested checkpoint path (against output_dir) and then requires the result to be an existing directory containing checkpoint metadata. If resolution yields a path that is not a directory — typo'd checkpoint name, rotated-away checkpoint (save_total_limit deleted it), or a file path — a ValueError is raised before any state is touched.","triggerScenarios":"Calling resume('checkpoint-500') when that step was never saved or was deleted by rotate_checkpoints due to save_total_limit; resuming an auto-resume path from a cleared/renamed output_dir; passing a path to a file instead of the checkpoint directory.","commonSituations":"save_total_limit rotated out the checkpoint the job tries to resume from; manual cleanup of output_dir between attempts; resume-from string copied from a different run's logs.","solutions":["List output_dir contents and resume from a checkpoint directory that actually exists (e.g. the highest-numbered remaining checkpoint-*)","Increase save_total_limit or disable rotation if you must keep the resume target","Pass the checkpoint directory path (not a file inside it) and verify with os.path.isdir first","If the checkpoint is gone, restart training from the base model instead of resuming"],"exampleFix":"# before\ncoordinator.resume(\"checkpoint-3000\")  # rotated away by save_total_limit=2\n\n# after\nimport os\nckpts = sorted((d for d in os.listdir(output_dir) if d.startswith(\"checkpoint-\")), key=lambda d: int(d.split(\"-\")[-1]))\ncoordinator.resume(ckpts[-1])  # newest surviving checkpoint","handlingStrategy":"validation","validationCode":"import os\n\ndef resolve_existing_checkpoint(output_dir: str, name: str):\n    ckpt = os.path.join(output_dir, name) if not os.path.isabs(name) else name\n    return ckpt if os.path.isdir(ckpt) else None","typeGuard":null,"tryCatchPattern":"try:\n    coordinator.resume(ckpt_path)\nexcept ValueError as e:\n    if \"does not exist\" in str(e):\n        ckpt = newest_surviving_checkpoint(output_dir)\n        coordinator.resume(ckpt) if ckpt else restart_from_base()","preventionTips":["Check os.path.isdir on the resolved checkpoint dir before calling resume","Size save_total_limit so the auto-resume target cannot be rotated away mid-run"],"tags":["checkpoint","resume","filesystem","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}