tracel-ai/burn · error

Can delete optimizer checkpoint.

Error message

Can delete optimizer checkpoint.

What it means

Same checkpointing flow as the model delete: the optimizer checkpointer's `delete(epoch)` is unwrapped with expect() and panics with this message when the optimizer checkpoint for that epoch cannot be deleted (file missing or IO error).

Solutions

1. Verify the optimizer checkpoint file for the epoch exists in the checkpoint directory and is writable.
2. Keep model/optimizer/scheduler checkpoints as an atomic set — restore or delete all three together.
3. Avoid external cleanup of individual checkpoint files while training is running.
4. Use a dedicated checkpoint directory per run to prevent concurrent deletion.
5. If your workflow deletes files externally, disable the Delete actions or make your wrapper tolerate missing epochs.

Example fix

// before: deleting epoch 12 after manual cleanup removed only optim-12
// after: delete the whole epoch as a unit or check first
for prefix in ["model", "optim", "scheduler"] {
    assert!(Path::new(dir).join(format!("{prefix}-{epoch}")).exists(), "{prefix} checkpoint missing");
}
checkpointer.checkpoint(actions);

let optim_path = Path::new(&checkpoint_dir).join(format!("optim-{epoch}"));
assert!(optim_path.exists(), "optimizer checkpoint for epoch {epoch} not found before delete");

fn has_full_checkpoint(dir: &str, epoch: usize) -> bool {
    ["model", "optim", "scheduler"].iter().all(|p| Path::new(dir).join(format!("{p}-{epoch}")).exists())
}

let result = std::panic::catch_unwind(AssertUnwindSafe(|| checkpointer.checkpoint(actions)));
if result.is_err() { /* restore or skip */ }

When it happens

Trigger: Applying CheckpointingAction::Delete(epoch) where the optimizer checkpoint file for `epoch` is absent, already removed, or the recorder lacks permission/disk access to remove it.

Common situations: Partial checkpoint sets (model file present but optimizer file deleted by disk cleanup or an interrupted save); wrong checkpoint directory; concurrent jobs deleting each other's files.

Related errors

• Can save optimizer checkpoint.
• Can delete learning rate scheduler checkpoint.
• Can delete model checkpoint.
• Can load optimizer checkpoint.
• Can save learning rate scheduler checkpoint.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/03419bb2bffe0801. Report an issue: GitHub.

Appendix: source

Thrown at crates/burn-train/src/learner/base.rs:170

            lr_scheduler,
            strategy,
            _phantom: PhantomData,
        }
    }

    /// Create checkpoint for the training process.
    pub fn checkpoint(&mut self, learner: &Learner<M>, epoch: usize, store: &EventStoreClient) {
        let actions = self.strategy.checkpointing(epoch, store);

        for action in actions {
            match action {
                CheckpointingAction::Delete(epoch) => {
                    self.model
                        .delete(epoch)
                        .expect("Can delete model checkpoint.");
                    self.optim
                        .delete(epoch)
                        .expect("Can delete optimizer checkpoint.");
                    self.lr_scheduler
                        .delete(epoch)
                        .expect("Can delete learning rate scheduler checkpoint.");
                }
                CheckpointingAction::Save => {
                    self.model
                        .save(epoch, learner.model.clone().into_record())
                        .expect("Can save model checkpoint.");
                    self.optim
                        .save(epoch, learner.optim.to_record())
                        .expect("Can save optimizer checkpoint.");
                    self.lr_scheduler
                        .save(epoch, learner.lr_scheduler.to_record())
                        .expect("Can save learning rate scheduler checkpoint.");
                }
            }
        }
    }

View on GitHub (pinned to d16f7ba2ed)