Lightning-AI/pytorch-lightning · warning

The dirpath has changed from {dirpath_from_ckpt!r} to {self.

Error message

The dirpath has changed from {dirpath_from_ckpt!r} to {self.dirpath!r}, therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded.

What it means

When ModelCheckpoint.load_state_dict restores from a checkpoint whose dirpath differs from the current callback's dirpath, only best_model_path is restored; the other state (best_model_score, kth_best_model_path, kth_value, last_model_path, best_k_models) is dropped because those files live in the old directory.

Source

Thrown at src/lightning/pytorch/callbacks/model_checkpoint.py:566

            "dirpath": self.dirpath,
            "best_k_models": self.best_k_models,
            "kth_best_model_path": self.kth_best_model_path,
            "kth_value": self.kth_value,
            "last_model_path": self.last_model_path,
        }

    @override
    def load_state_dict(self, state_dict: dict[str, Any]) -> None:
        dirpath_from_ckpt = state_dict.get("dirpath", self.dirpath)

        if self.dirpath == dirpath_from_ckpt:
            self.best_model_score = state_dict["best_model_score"]
            self.kth_best_model_path = state_dict.get("kth_best_model_path", self.kth_best_model_path)
            self.kth_value = state_dict.get("kth_value", self.kth_value)
            self.best_k_models = state_dict.get("best_k_models", self.best_k_models)
            self.last_model_path = state_dict.get("last_model_path", self.last_model_path)
        else:
            warnings.warn(
                f"The dirpath has changed from {dirpath_from_ckpt!r} to {self.dirpath!r},"
                " therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and"
                " `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded."
            )

        self.best_model_path = state_dict["best_model_path"]

    def _save_topk_checkpoint(self, trainer: "pl.Trainer", monitor_candidates: dict[str, Tensor]) -> None:
        if self.save_top_k == 0:
            return

        # validate metric
        if self.monitor is not None:
            if self.monitor not in monitor_candidates:
                m = (
                    f"`ModelCheckpoint(monitor={self.monitor!r})` could not find the monitored key in the returned"
                    f" metrics: {list(monitor_candidates)}."
                    f" HINT: Did you call `log({self.monitor!r}, value)` in the `LightningModule`?"

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Keep the same dirpath when resuming: reuse the ModelCheckpoint config from the original run
2. Or read the old checkpoint's callback_state and manually restore best_model_score etc.
3. Or treat resume as a fresh checkpoint lineage and accept the reset

Example fix

# before
ckpt = ModelCheckpoint(dirpath=f'runs/{timestamp}')  # changes every run
trainer.fit(model, ckpt_path='runs/old/last.ckpt', callbacks=[ckpt])
# after
ckpt = ModelCheckpoint(dirpath='runs/exp1')
trainer.fit(model, ckpt_path='runs/exp1/last.ckpt', callbacks=[ckpt])

import torch
ckpt = torch.load(ckpt_path, map_location='cpu', weights_only=False)
state = next(c for c in ckpt['callbacks'] if 'ModelCheckpoint' in c)
assert state.get('dirpath') == mc.dirpath, 'dirpath changed; state will not fully restore'

When it happens

Trigger: Resuming training with Trainer(plugins=[ModelCheckpoint(dirpath='new/dir')], ...) from a ckpt_path saved with a different dirpath (or after changing default_root_dir/CV options).

Common situations: Changing output directories between runs, resume scripts with timestamped dirpaths, hyperparameter sweeps reusing callbacks.

Related errors

• `ModelCheckpoint(save_last='link')` is only supported for lo
• `ModelCheckpoint(monitor={self.monitor!r})` could not find t
• Invalid value for save_top_k={self.save_top_k}. Must be >= -
• `.{fn}(ckpt_path="best")` is set but `ModelCheckpoint` is no
• `.{fn}(ckpt_path="hpc")` is set but no HPC checkpoint was fo

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/54ef4a10d0d09ac7. Report an issue: GitHub.