Lightning-AI/pytorch-lightning · error · MisconfigurationException

`ModelCheckpoint(monitor={self.monitor!r})` could not find t

Error message

`ModelCheckpoint(monitor={self.monitor!r})` could not find the monitored key in the returned metrics: {list(monitor_candidates)}. HINT: Did you call `log({self.monitor!r}, value)` in the `LightningModule`?

What it means

When `save_top_k >= 1` and a monitor is set, ModelCheckpoint looks for the monitored key among the metrics it can see. If missing, and validation has already run at least once (`val_loop._has_run`), it raises MisconfigurationException with a hint to log the key; otherwise it only warns once.

Source

Thrown at src/lightning/pytorch/callbacks/model_checkpoint.py:587

                " `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded."
            )

        self.best_model_path = state_dict["best_model_path"]

    def _save_topk_checkpoint(self, trainer: "pl.Trainer", monitor_candidates: dict[str, Tensor]) -> None:
        if self.save_top_k == 0:
            return

        # validate metric
        if self.monitor is not None:
            if self.monitor not in monitor_candidates:
                m = (
                    f"`ModelCheckpoint(monitor={self.monitor!r})` could not find the monitored key in the returned"
                    f" metrics: {list(monitor_candidates)}."
                    f" HINT: Did you call `log({self.monitor!r}, value)` in the `LightningModule`?"
                )
                if trainer.fit_loop.epoch_loop.val_loop._has_run:
                    raise MisconfigurationException(m)
                warning_cache.warn(m)
            self._save_monitor_checkpoint(trainer, monitor_candidates)
        else:
            self._save_none_monitor_checkpoint(trainer, monitor_candidates)

    def _save_checkpoint(self, trainer: "pl.Trainer", filepath: str) -> None:
        """Save the checkpoint to the given filepath.

        For manual optimization, we rely on the fact that the model's training_step method saves the model state before
        the optimizer step, so we can use that state directly.

        """
        trainer.save_checkpoint(filepath, self.save_weights_only)
        self._last_global_step_saved = trainer.global_step
        self._last_checkpoint_saved = filepath

        # notify loggers
        if trainer.is_global_zero:

View on GitHub (pinned to 9fed5c27d2)

Solutions

Match names exactly: log `self.log('val_f1', ...)` in validation_step (with on_epoch=True or in validation_epoch_end) and set monitor='val_f1'
Check the list of available metric keys printed in the error and use one of them
If validation hasn't run yet, note it only warns the first time — ensure the metric exists before the first checkpointing event

Example fix

# before
# monitor='val_f1' but module logs 'val_acc'
ModelCheckpoint(monitor='val_f1')
# after
def validation_step(self, batch, batch_idx):
    self.log('val_f1', f1, prog_bar=True)
ModelCheckpoint(monitor='val_f1')

Defensive patterns

Strategy: validation

Validate before calling

# smoke check: run 1 val epoch and confirm the monitor key appears
# assert monitor in trainer.callback_metrics after a sanity run
monitor = 'val_loss'
# after trainer.validate / a 1-step fit:
# assert monitor in trainer.callback_metrics

Try / catch

try:
    trainer.fit(model)
except MisconfigurationException as e:
    if 'could not find the monitored key' in str(e):
        avail = eval(e.args[0].split('metrics: ')[1].split('.')[0])  # or parse manually
        # choose a corrected monitor from avail and retry
        raise

Prevention

Define metric names as module-level constants used by both self.log and ModelCheckpoint
Run a fast smoke fit (limit_train_batches=1, limit_val_batches=1) before long runs
Cross-check logged keys via logger output or trainer.callback_metrics

When it happens

Trigger: `ModelCheckpoint(monitor='val_f1')` but the module never calls `self.log('val_f1', ...)`; metric logged only in training while checkpointing expects a validation metric; monitor name typo; metric logged with `sync_dist` conditions so it's absent on some ranks.

Common situations: Renaming logged metrics and forgetting the checkpoint monitor; using a different logger prefix ('val/f1' vs 'val_f1'); logging inside `on_validation_epoch_end` with add_n_metrics misconfigured; logging on epoch vs step mismatch.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/ebea82b17da6d912. Report an issue: GitHub.