huggingface/transformers · error · ValueError

[deepspeed] failed to resume from checkpoint {checkpoint_pat

Error message

[deepspeed] failed to resume from checkpoint {checkpoint_path}

What it means

Error "[deepspeed] failed to resume from checkpoint {checkpoint_path}" thrown in huggingface/transformers.

Source

Thrown at src/transformers/integrations/deepspeed.py:669

    # it's possible that the user is trying to resume from model_path, which doesn't necessarily
    # contain a deepspeed checkpoint. e.g. examples just check if the dir exists and assume it's
    # a resume from a checkpoint and not just a local pretrained weight. So we check here if the
    # path contains what looks like a deepspeed checkpoint
    import glob

    deepspeed_checkpoint_dirs = sorted(glob.glob(f"{checkpoint_path}/global_step*"))

    if len(deepspeed_checkpoint_dirs) > 0:
        logger.info(f"Attempting to resume from {checkpoint_path}")
        # this magically updates self.optimizer and self.lr_scheduler
        load_path, _ = deepspeed_engine.load_checkpoint(
            checkpoint_path,
            load_module_strict=load_module_strict,
            load_optimizer_states=True,
            load_lr_scheduler_states=True,
        )
        if load_path is None:
            raise ValueError(f"[deepspeed] failed to resume from checkpoint {checkpoint_path}")
    else:
        raise ValueError(f"Can't find a valid checkpoint at {checkpoint_path}")


def propagate_args_to_deepspeed(accelerator, args, auto_find_batch_size=False):
    """
    Sets values in the deepspeed plugin based on the TrainingArguments.

    Args:
        accelerator (`Accelerator`): The Accelerator object.
        args (`TrainingArguments`): The training arguments to propagate to DeepSpeed config.
        auto_find_batch_size (`bool`, *optional*, defaults to `False`):
            Whether batch size was auto-discovered by trying increasingly smaller sizes.
    """
    ds_plugin = accelerator.state.deepspeed_plugin

    ds_plugin.hf_ds_config = HfTrainerDeepSpeedConfig(ds_plugin.hf_ds_config.config)
    ds_plugin.deepspeed_config = ds_plugin.hf_ds_config.config

View on GitHub (pinned to a597f97485)

Solutions

1. Verify the checkpoint path contains a valid DeepSpeed checkpoint.
2. Re-run saving to produce a complete checkpoint before resuming.

When it happens

Trigger: Raised when DeepSpeed fails to resume training from the given checkpoint path.

Common situations: Corrupted, incomplete, or mismatched DeepSpeed checkpoint passed via resume_from_checkpoint.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/88adc2499d64c870. Report an issue: GitHub.