tracel-ai/burn · error

Can save model checkpoint.

Error message

Can save model checkpoint.

What it means

On CheckpointingAction::Save, the learner saves the model record via the model checkpointer; the result is unwrapped with expect(). A failure writing the model checkpoint file (disk full, permission denied, serialization error) panics with this message.

Source

Thrown at crates/burn-train/src/learner/base.rs:178

        let actions = self.strategy.checkpointing(epoch, store);

        for action in actions {
            match action {
                CheckpointingAction::Delete(epoch) => {
                    self.model
                        .delete(epoch)
                        .expect("Can delete model checkpoint.");
                    self.optim
                        .delete(epoch)
                        .expect("Can delete optimizer checkpoint.");
                    self.lr_scheduler
                        .delete(epoch)
                        .expect("Can delete learning rate scheduler checkpoint.");
                }
                CheckpointingAction::Save => {
                    self.model
                        .save(epoch, learner.model.clone().into_record())
                        .expect("Can save model checkpoint.");
                    self.optim
                        .save(epoch, learner.optim.to_record())
                        .expect("Can save optimizer checkpoint.");
                    self.lr_scheduler
                        .save(epoch, learner.lr_scheduler.to_record())
                        .expect("Can save learning rate scheduler checkpoint.");
                }
            }
        }
    }

    /// Load a training checkpoint.
    ///
    /// No device is taken: checkpoints are device-free burnpack records (file-backed bytes). On
    /// load, the model keeps the device of the learner's existing parameters, and the optimizer
    /// state is migrated to each parameter's device on the next step. The training device is fixed
    /// earlier, when the learner's model is created/forked.
    pub fn load_checkpoint(&self, mut learner: Learner<M>, epoch: usize) -> Learner<M> {

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Check free disk space (df) and clean space or point the checkpointer to a larger volume.
2. Verify the checkpoint directory exists and is writable by the training process (permissions, mount flags).
3. Validate the recorder settings (path, file format) used to construct the FileCheckpointer.
4. Reduce checkpoint size/frequency or save to compressed storage if disk pressure is chronic.
5. Write checkpoints to a local temp dir and sync to durable storage after success.

Example fix

// before
let model_checkpointer = FileCheckpointer::new(Recorder, "/mnt/ro-checkpoints", "model"); // read-only mount
// after
let dir = "/checkpoints";
std::fs::create_dir_all(dir).expect("checkpoint dir must be writable");
assert!(test_write(dir), "checkpoint dir not writable");
let model_checkpointer = FileCheckpointer::new(Recorder, dir, "model");

let dir = Path::new(&checkpoint_dir);
std::fs::create_dir_all(dir).expect("create checkpoint dir");
let probe = dir.join(".write_probe");
std::fs::write(&probe, b"ok").expect("checkpoint dir not writable");
std::fs::remove_file(&probe).ok();

fn checkpoint_dir_writable(dir: &str) -> bool {
    let p = Path::new(dir).join(".probe");
    std::fs::write(&p, b"1").is_ok() && std::fs::remove_file(p).is_ok()
}

let result = std::panic::catch_unwind(AssertUnwindSafe(|| checkpointer.checkpoint(actions)));
if result.is_err() { eprintln!("model checkpoint save failed; check disk/permissions"); }

When it happens

Trigger: Calling checkpoint() with a Save action when the model record cannot be serialized or written: full disk, read-only filesystem, invalid/unwritable checkpoint directory, recorder serialization failure (e.g. corrupted target, unsupported format).

Common situations: Training on a node whose scratch disk filled up; checkpoint dir mounted read-only or deleted mid-run; container with restricted write permissions; using a recorder format that can't encode the model record.

Related errors

• Can save optimizer checkpoint.
• Failed to load module from file
• Can delete model checkpoint.
• Can delete optimizer checkpoint.
• Can delete learning rate scheduler checkpoint.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/27edcc848b620ecb. Report an issue: GitHub.