Lightning-AI/pytorch-lightning · error · ValueError

`.{fn}(ckpt_path="hpc")` is set but no HPC checkpoint was fo

Error message

`.{fn}(ckpt_path="hpc")` is set but no HPC checkpoint was found. Please pass an exact checkpoint path to `.{fn}(ckpt_path=...)`

What it means

Raised by _parse_ckpt_path when ckpt_path="hpc" is requested but the connector found no HPC checkpoint. HPC checkpoints are only discovered when the connector's hpc_resume_path was set (e.g., saved during preemption/interruption handling in a supported environment), so in normal runs there is nothing to resume.

Source

Thrown at src/lightning/pytorch/trainer/connectors/checkpoint_connector.py:199

        elif ckpt_path == "last":
            candidates = {getattr(ft, "ckpt_path", None) for ft in ft_checkpoints}
            for callback in self.trainer.checkpoint_callbacks:
                if isinstance(callback, ModelCheckpoint):
                    candidates |= callback._find_last_checkpoints(self.trainer)
            candidates_fs = {path: get_filesystem(path) for path in candidates if path}
            candidates_ts = {path: fs.modified(path) for path, fs in candidates_fs.items() if fs.exists(path)}
            if not candidates_ts:
                # not an error so it can be set and forget before the first `fit` run
                rank_zero_warn(
                    f'.{fn}(ckpt_path="last") is set, but there is no last checkpoint available.'
                    " No checkpoint will be loaded. HINT: Set `ModelCheckpoint(..., save_last=True)`."
                )
                return None
            ckpt_path = max(candidates_ts, key=candidates_ts.get)  # type: ignore[arg-type]

        elif ckpt_path == "hpc":
            if not self._hpc_resume_path:
                raise ValueError(
                    f'`.{fn}(ckpt_path="hpc")` is set but no HPC checkpoint was found.'
                    f" Please pass an exact checkpoint path to `.{fn}(ckpt_path=...)`"
                )
            ckpt_path = self._hpc_resume_path

        elif _is_registry(ckpt_path) and module_available("litmodels"):
            ckpt_path = find_model_local_ckpt_path(
                ckpt_path,
                default_model_registry=self.trainer._model_registry,
                default_root_dir=self.trainer.default_root_dir,
            )

        if not ckpt_path:
            raise ValueError(
                f"`.{fn}()` found no path for the best weights: {ckpt_path!r}. Please"
                f" specify a path for a checkpoint `.{fn}(ckpt_path=PATH)`"
            )
        return ckpt_path

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Pass the exact checkpoint path instead: trainer.fit(ckpt_path="path/to/last.ckpt")
2. Use ckpt_path="last" to pick the most recent non-HPC checkpoint, or "best" for the best one
3. If resuming after preemption is intended, verify the HPC checkpoint files exist and the environment (SLURM signal handling) actually triggered the save

Example fix

# before
trainer.fit(model, ckpt_path="hpc")
# after
trainer.fit(model, ckpt_path="lightning_logs/version_0/checkpoints/last.ckpt")

if ckpt_path == "hpc":
    # only valid in preemption-enabled environments; prefer explicit path
    ckpt_path = trainer.checkpoint_connector._hpc_resume_path or "last"

When it happens

Trigger: trainer.fit(ckpt_path="hpc") (or validate/test/predict) when no HPC interruption checkpoint exists — i.e., the run was never preempted or the HPC save files (e.g., saved at crash time) are absent from the resume location.

Common situations: Reusing resume boilerplate copied from SLURM/preemption workflows on a normal run; after manually cleaning the output directory; expecting "hpc" to behave like "last".

Related errors

• `.{fn}()` found no path for the best weights: {ckpt_path!r}.
• Trying to restore optimizer state but checkpoint contains on
• Trying to restore learning rate scheduler state but checkpoi
• Could not find a distributed model in the provided checkpoin
• Found multiple distributed models in the given state. Loadin

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/042410875df96a3e. Report an issue: GitHub.