{"record":{"id":"d46a3ac9838c7fb3","repo":"tracel-ai/burn","slug":"can-save-optimizer-checkpoint","errorCode":null,"errorMessage":"Can save optimizer checkpoint.","messagePattern":"Can save optimizer checkpoint\\.","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/burn-train/src/learner/base.rs","lineNumber":181,"sourceCode":"            match action {\n                CheckpointingAction::Delete(epoch) => {\n                    self.model\n                        .delete(epoch)\n                        .expect(\"Can delete model checkpoint.\");\n                    self.optim\n                        .delete(epoch)\n                        .expect(\"Can delete optimizer checkpoint.\");\n                    self.lr_scheduler\n                        .delete(epoch)\n                        .expect(\"Can delete learning rate scheduler checkpoint.\");\n                }\n                CheckpointingAction::Save => {\n                    self.model\n                        .save(epoch, learner.model.clone().into_record())\n                        .expect(\"Can save model checkpoint.\");\n                    self.optim\n                        .save(epoch, learner.optim.to_record())\n                        .expect(\"Can save optimizer checkpoint.\");\n                    self.lr_scheduler\n                        .save(epoch, learner.lr_scheduler.to_record())\n                        .expect(\"Can save learning rate scheduler checkpoint.\");\n                }\n            }\n        }\n    }\n\n    /// Load a training checkpoint.\n    ///\n    /// No device is taken: checkpoints are device-free burnpack records (file-backed bytes). On\n    /// load, the model keeps the device of the learner's existing parameters, and the optimizer\n    /// state is migrated to each parameter's device on the next step. The training device is fixed\n    /// earlier, when the learner's model is created/forked.\n    pub fn load_checkpoint(&self, mut learner: Learner<M>, epoch: usize) -> Learner<M> {\n        let record = self\n            .model\n            .restore(epoch)","sourceCodeStart":163,"sourceCodeEnd":199,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/learner/base.rs#L163-L199","documentation":"On CheckpointingAction::Save, the optimizer state record is written through the optimizer checkpointer and unwrapped with expect(). A write/serialization failure for the optimizer checkpoint panics with this message.","triggerScenarios":"Calling checkpoint() with Save when the optimizer record cannot be saved: disk full, unwritable directory, or serialization failure of the optimizer state record (which can be large — moments + moments buffers — making it more likely to exhaust disk than the model file).","commonSituations":"Adam/AdamW states on large models make optimizer checkpoints huge and disk fills mid-run; checkpoint dir permissions changed; interrupted prior save left a corrupt target; network mount flaked during write.","solutions":["Check free disk space; optimizer state can be several times the model size — allocate accordingly.","Confirm the optimizer checkpointer's directory exists and is writable.","Delete stale optimizer checkpoints of old epochs to reclaim space before the save.","If a prior failed save left partial files, remove the epoch's optimizer checkpoint and retry.","Consider saving optimizer state less frequently than model weights if disk is constrained."],"exampleFix":"// before\noptim.save(epoch, learner.optim.to_record()).expect(\"Can save optimizer checkpoint.\"); // ENOSPC\n// after: preflight space check\nlet needed = estimated_optim_state_bytes;\nassert!(free_space(dir) > needed, \"not enough disk for optimizer checkpoint\");\noptim.save(epoch, learner.optim.to_record()).expect(\"Can save optimizer checkpoint.\");","handlingStrategy":"validation","validationCode":"let free = fs_free_bytes(&checkpoint_dir);\nlet needed = 3 * model_size_bytes; // AdamW keeps ~2 extra state buffers\nassert!(free > needed, \"insufficient disk for optimizer checkpoint: {free} free\");","typeGuard":"fn has_disk_space(dir: &str, min_bytes: u64) -> bool {\n    fs2::available_space(dir).map(|f| f > min_bytes).unwrap_or(false)\n}","tryCatchPattern":"let result = std::panic::catch_unwind(AssertUnwindSafe(|| checkpointer.checkpoint(actions)));\nif result.is_err() { /* free space, prune old epochs, retry */ }","preventionTips":["Budget disk for optimizer state (2-3x model size for adaptive optimizers like Adam).","Prune old epochs with keepN/keepOld settings instead of manual deletion.","Monitor disk usage in your training loop and stop before ENOSPC.","Write optimizer checkpoints less frequently if disk is tight."],"tags":["rust","panic","io","checkpoint","optimizer-state","disk-full"],"backgroundTag":"checkpoint-io-failure","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}