tracel-ai/burn · error

Can load optimizer checkpoint.

Error message

Can load optimizer checkpoint.

What it means

In `load_checkpoint`, the optimizer record is restored via the optimizer checkpointer's `restore(epoch)` and unwrapped with expect(). A missing or unreadable optimizer checkpoint for the epoch panics with this message.

Source

Thrown at crates/burn-train/src/learner/base.rs:206

    }

    /// Load a training checkpoint.
    ///
    /// No device is taken: checkpoints are device-free burnpack records (file-backed bytes). On
    /// load, the model keeps the device of the learner's existing parameters, and the optimizer
    /// state is migrated to each parameter's device on the next step. The training device is fixed
    /// earlier, when the learner's model is created/forked.
    pub fn load_checkpoint(&self, mut learner: Learner<M>, epoch: usize) -> Learner<M> {
        let record = self
            .model
            .restore(epoch)
            .expect("Can load model checkpoint.");
        learner.load_model(record);

        let record = self
            .optim
            .restore(epoch)
            .expect("Can load optimizer checkpoint.");
        learner.load_optim(record);

        let record = self
            .lr_scheduler
            .restore(epoch)
            .expect("Can load learning rate scheduler checkpoint.");
        learner.load_scheduler(record);

        learner
    }
}

/// Cloneable reference to an early stopping strategy
pub(crate) type EarlyStoppingStrategyRef = Box<dyn CloneEarlyStoppingStrategy>;

#[derive(Clone, Default)]
/// A handle that allows aborting the training/evaluation process early.
pub struct Interrupter {

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Verify the optimizer checkpoint file for the epoch exists in the optimizer checkpointer's directory.
2. Point the optimizer checkpointer at the same directory/prefix used by the saving run.
3. If the optimizer type or burn version changed, re-match it or start training fresh (or accept degraded resume from weights only via your own load path).
4. Keep model/optimizer/scheduler saves atomic so a crash can't leave only model files.
5. Restore the missing optimizer file from backup if you have periodic storage snapshots.

Example fix

// before
let optim_cp = FileCheckpointer::new(Recorder, "/other/dir", "optim"); // wrong dir on resume
// after
let base = "/checkpoints/run-1"; // same as the saving run
let optim_cp = FileCheckpointer::new(Recorder, base, "optim");
learner::load_checkpoint(learner, epoch);

let optim_path = Path::new(&checkpoint_dir).join(format!("optim-{epoch}"));
assert!(optim_path.exists(), "optimizer checkpoint missing for epoch {epoch}");
// also confirm the optimizer type matches the one used at save time
assert_eq!(current_optimizer_name, saved_optimizer_name);

fn optimizer_checkpoint_exists(dir: &str, epoch: usize) -> bool {
    Path::new(dir).join(format!("optim-{epoch}")).exists()
}

let result = std::panic::catch_unwind(AssertUnwindSafe(|| learner::load_checkpoint(learner, epoch)));
if result.is_err() { eprintln!("optimizer restore failed; resuming with fresh optimizer state"); }

When it happens

Trigger: Calling load_checkpoint(learner, epoch) where the model checkpoint exists but the optimizer checkpoint for that epoch does not (partial save from a crash, external cleanup removed optim files) or fails to deserialize (optimizer changed between runs, e.g. switched Adam to SGD, or version mismatch).

Common situations: Model and optimizer checkpointers pointed at different directories on resume; disk cleanup deleted the large optimizer files but kept model weights; changing the optimizer type while resuming.

Related errors

• Can load model checkpoint.
• Optimizer record tensors should carry a parameter id.
• Can delete optimizer checkpoint.
• Can save optimizer checkpoint.
• Can load learning rate scheduler checkpoint.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/b95888c2481d62bf. Report an issue: GitHub.