hiyouga/LlamaFactory · error · ValueError

Checkpoint directory does not exist: {ckpt_dir}

Error message

Checkpoint directory does not exist: {ckpt_dir}

What it means

TrainingCheckpointCoordinator.resume() resolves the requested checkpoint path (against output_dir) and then requires the result to be an existing directory containing checkpoint metadata. If resolution yields a path that is not a directory — typo'd checkpoint name, rotated-away checkpoint (save_total_limit deleted it), or a file path — a ValueError is raised before any state is touched.

Source

Thrown at src/llamafactory/v1/core/utils/checkpoint.py:301

            save_rng_state(ckpt_dir, rank)

        DistributedInterface().sync()

        if rank == 0:
            mark_checkpoint_complete(ckpt_dir)
            if self._t.args.save_total_limit is not None:
                rotate_checkpoints(self._t.args.output_dir, self._t.args.save_total_limit)

        logger.info_rank0(f"Checkpoint saved to {ckpt_dir}")

    def resume(self, ckpt_path: str) -> None:
        """Restore full training state from a checkpoint directory."""
        ckpt_dir = resolve_resume_checkpoint_path(ckpt_path, self._t.args.output_dir)
        if ckpt_dir is None:
            return

        if not os.path.isdir(ckpt_dir):
            raise ValueError(f"Checkpoint directory does not exist: {ckpt_dir}")

        rank = DistributedInterface().get_rank()

        metadata = load_metadata(ckpt_dir)
        self._t.global_step = metadata["global_step"]
        self._t._resume_epoch = metadata["epoch"]

        if self._dist_name in ("fsdp2", "fsdpturbo", "deepspeed"):
            from ...plugins.trainer_plugins.distributed.interface import DistributedPlugin

            DistributedPlugin(self._dist_name).load_checkpoint(
                self._t.model,
                self._t.optimizer,
                ckpt_dir,
                processor=self._t.renderer.processor,
            )
        else:
            _load_standard_training_states(

View on GitHub (pinned to f28afaf635)

Solutions

1. List output_dir contents and resume from a checkpoint directory that actually exists (e.g. the highest-numbered remaining checkpoint-*)
2. Increase save_total_limit or disable rotation if you must keep the resume target
3. Pass the checkpoint directory path (not a file inside it) and verify with os.path.isdir first
4. If the checkpoint is gone, restart training from the base model instead of resuming

Example fix

# before
coordinator.resume("checkpoint-3000")  # rotated away by save_total_limit=2

# after
import os
ckpts = sorted((d for d in os.listdir(output_dir) if d.startswith("checkpoint-")), key=lambda d: int(d.split("-")[-1]))
coordinator.resume(ckpts[-1])  # newest surviving checkpoint

import os

def resolve_existing_checkpoint(output_dir: str, name: str):
    ckpt = os.path.join(output_dir, name) if not os.path.isabs(name) else name
    return ckpt if os.path.isdir(ckpt) else None

try:
    coordinator.resume(ckpt_path)
except ValueError as e:
    if "does not exist" in str(e):
        ckpt = newest_surviving_checkpoint(output_dir)
        coordinator.resume(ckpt) if ckpt else restart_from_base()

When it happens

Trigger: Calling resume('checkpoint-500') when that step was never saved or was deleted by rotate_checkpoints due to save_total_limit; resuming an auto-resume path from a cleared/renamed output_dir; passing a path to a file instead of the checkpoint directory.

Common situations: save_total_limit rotated out the checkpoint the job tries to resume from; manual cleanup of output_dir between attempts; resume-from string copied from a different run's logs.

Related errors

• `save_dir` already exists, use another one.
• Output directory already exists and is not empty. Please set
• `resume_from_checkpoint` will be supported in the future ver
• Unexpected missing keys when loading checkpoint model weight
• No checkpoint files found in {hf_model_path}

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/dda47abd0a0c6947. Report an issue: GitHub.