huggingface/transformers · error · ValueError

Can't find a valid checkpoint at {checkpoint_path}

Error message

Can't find a valid checkpoint at {checkpoint_path}

What it means

Error "Can't find a valid checkpoint at {checkpoint_path}" thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepspeed.py:671

    # a resume from a checkpoint and not just a local pretrained weight. So we check here if the
    # path contains what looks like a deepspeed checkpoint
    import glob

    deepspeed_checkpoint_dirs = sorted(glob.glob(f"{checkpoint_path}/global_step*"))

    if len(deepspeed_checkpoint_dirs) > 0:
        logger.info(f"Attempting to resume from {checkpoint_path}")
        # this magically updates self.optimizer and self.lr_scheduler
        load_path, _ = deepspeed_engine.load_checkpoint(
            checkpoint_path,
            load_module_strict=load_module_strict,
            load_optimizer_states=True,
            load_lr_scheduler_states=True,
        )
        if load_path is None:
            raise ValueError(f"[deepspeed] failed to resume from checkpoint {checkpoint_path}")
    else:
        raise ValueError(f"Can't find a valid checkpoint at {checkpoint_path}")


def propagate_args_to_deepspeed(accelerator, args, auto_find_batch_size=False):
    """
    Sets values in the deepspeed plugin based on the TrainingArguments.

    Args:
        accelerator (`Accelerator`): The Accelerator object.
        args (`TrainingArguments`): The training arguments to propagate to DeepSpeed config.
        auto_find_batch_size (`bool`, *optional*, defaults to `False`):
            Whether batch size was auto-discovered by trying increasingly smaller sizes.
    """
    ds_plugin = accelerator.state.deepspeed_plugin

    ds_plugin.hf_ds_config = HfTrainerDeepSpeedConfig(ds_plugin.hf_ds_config.config)
    ds_plugin.deepspeed_config = ds_plugin.hf_ds_config.config
    ds_plugin.hf_ds_config.trainer_config_process(args, auto_find_batch_size)

View on GitHub (pinned to a597f97485)

Solutions

1. Point `load_path` to a directory containing a valid DeepSpeed checkpoint (with the `latest` file).

When it happens

Trigger: Raised when no valid DeepSpeed checkpoint files exist at the given checkpoint path.

Common situations: Wrong checkpoint directory or a directory missing the DeepSpeed checkpoint tags/files.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/3c0ec23df3d583c7. Report an issue: GitHub.