tracel-ai/burn · error

Can save optimizer checkpoint.

Error message

Can save optimizer checkpoint.

What it means

On CheckpointingAction::Save, the optimizer state record is written through the optimizer checkpointer and unwrapped with expect(). A write/serialization failure for the optimizer checkpoint panics with this message.

Source

Thrown at crates/burn-train/src/learner/base.rs:181

            match action {
                CheckpointingAction::Delete(epoch) => {
                    self.model
                        .delete(epoch)
                        .expect("Can delete model checkpoint.");
                    self.optim
                        .delete(epoch)
                        .expect("Can delete optimizer checkpoint.");
                    self.lr_scheduler
                        .delete(epoch)
                        .expect("Can delete learning rate scheduler checkpoint.");
                }
                CheckpointingAction::Save => {
                    self.model
                        .save(epoch, learner.model.clone().into_record())
                        .expect("Can save model checkpoint.");
                    self.optim
                        .save(epoch, learner.optim.to_record())
                        .expect("Can save optimizer checkpoint.");
                    self.lr_scheduler
                        .save(epoch, learner.lr_scheduler.to_record())
                        .expect("Can save learning rate scheduler checkpoint.");
                }
            }
        }
    }

    /// Load a training checkpoint.
    ///
    /// No device is taken: checkpoints are device-free burnpack records (file-backed bytes). On
    /// load, the model keeps the device of the learner's existing parameters, and the optimizer
    /// state is migrated to each parameter's device on the next step. The training device is fixed
    /// earlier, when the learner's model is created/forked.
    pub fn load_checkpoint(&self, mut learner: Learner<M>, epoch: usize) -> Learner<M> {
        let record = self
            .model
            .restore(epoch)

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Check free disk space; optimizer state can be several times the model size — allocate accordingly.
2. Confirm the optimizer checkpointer's directory exists and is writable.
3. Delete stale optimizer checkpoints of old epochs to reclaim space before the save.
4. If a prior failed save left partial files, remove the epoch's optimizer checkpoint and retry.
5. Consider saving optimizer state less frequently than model weights if disk is constrained.

Example fix

// before
optim.save(epoch, learner.optim.to_record()).expect("Can save optimizer checkpoint."); // ENOSPC
// after: preflight space check
let needed = estimated_optim_state_bytes;
assert!(free_space(dir) > needed, "not enough disk for optimizer checkpoint");
optim.save(epoch, learner.optim.to_record()).expect("Can save optimizer checkpoint.");

let free = fs_free_bytes(&checkpoint_dir);
let needed = 3 * model_size_bytes; // AdamW keeps ~2 extra state buffers
assert!(free > needed, "insufficient disk for optimizer checkpoint: {free} free");

fn has_disk_space(dir: &str, min_bytes: u64) -> bool {
    fs2::available_space(dir).map(|f| f > min_bytes).unwrap_or(false)
}

let result = std::panic::catch_unwind(AssertUnwindSafe(|| checkpointer.checkpoint(actions)));
if result.is_err() { /* free space, prune old epochs, retry */ }

When it happens

Trigger: Calling checkpoint() with Save when the optimizer record cannot be saved: disk full, unwritable directory, or serialization failure of the optimizer state record (which can be large — moments + moments buffers — making it more likely to exhaust disk than the model file).

Common situations: Adam/AdamW states on large models make optimizer checkpoints huge and disk fills mid-run; checkpoint dir permissions changed; interrupted prior save left a corrupt target; network mount flaked during write.

Related errors

• Can delete optimizer checkpoint.
• Can save model checkpoint.
• Failed to load module from file
• Can delete model checkpoint.
• Can delete learning rate scheduler checkpoint.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/d46a3ac9838c7fb3. Report an issue: GitHub.