{"record":{"id":"27edcc848b620ecb","repo":"tracel-ai/burn","slug":"can-save-model-checkpoint","errorCode":null,"errorMessage":"Can save model checkpoint.","messagePattern":"Can save model checkpoint\\.","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/burn-train/src/learner/base.rs","lineNumber":178,"sourceCode":"        let actions = self.strategy.checkpointing(epoch, store);\n\n        for action in actions {\n            match action {\n                CheckpointingAction::Delete(epoch) => {\n                    self.model\n                        .delete(epoch)\n                        .expect(\"Can delete model checkpoint.\");\n                    self.optim\n                        .delete(epoch)\n                        .expect(\"Can delete optimizer checkpoint.\");\n                    self.lr_scheduler\n                        .delete(epoch)\n                        .expect(\"Can delete learning rate scheduler checkpoint.\");\n                }\n                CheckpointingAction::Save => {\n                    self.model\n                        .save(epoch, learner.model.clone().into_record())\n                        .expect(\"Can save model checkpoint.\");\n                    self.optim\n                        .save(epoch, learner.optim.to_record())\n                        .expect(\"Can save optimizer checkpoint.\");\n                    self.lr_scheduler\n                        .save(epoch, learner.lr_scheduler.to_record())\n                        .expect(\"Can save learning rate scheduler checkpoint.\");\n                }\n            }\n        }\n    }\n\n    /// Load a training checkpoint.\n    ///\n    /// No device is taken: checkpoints are device-free burnpack records (file-backed bytes). On\n    /// load, the model keeps the device of the learner's existing parameters, and the optimizer\n    /// state is migrated to each parameter's device on the next step. The training device is fixed\n    /// earlier, when the learner's model is created/forked.\n    pub fn load_checkpoint(&self, mut learner: Learner<M>, epoch: usize) -> Learner<M> {","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/tracel-ai/burn/blob/d16f7ba2ed0d41408189384044cc886fb4c8f957/crates/burn-train/src/learner/base.rs#L160-L196","documentation":"On CheckpointingAction::Save, the learner saves the model record via the model checkpointer; the result is unwrapped with expect(). A failure writing the model checkpoint file (disk full, permission denied, serialization error) panics with this message.","triggerScenarios":"Calling checkpoint() with a Save action when the model record cannot be serialized or written: full disk, read-only filesystem, invalid/unwritable checkpoint directory, recorder serialization failure (e.g. corrupted target, unsupported format).","commonSituations":"Training on a node whose scratch disk filled up; checkpoint dir mounted read-only or deleted mid-run; container with restricted write permissions; using a recorder format that can't encode the model record.","solutions":["Check free disk space (df) and clean space or point the checkpointer to a larger volume.","Verify the checkpoint directory exists and is writable by the training process (permissions, mount flags).","Validate the recorder settings (path, file format) used to construct the FileCheckpointer.","Reduce checkpoint size/frequency or save to compressed storage if disk pressure is chronic.","Write checkpoints to a local temp dir and sync to durable storage after success."],"exampleFix":"// before\nlet model_checkpointer = FileCheckpointer::new(Recorder, \"/mnt/ro-checkpoints\", \"model\"); // read-only mount\n// after\nlet dir = \"/checkpoints\";\nstd::fs::create_dir_all(dir).expect(\"checkpoint dir must be writable\");\nassert!(test_write(dir), \"checkpoint dir not writable\");\nlet model_checkpointer = FileCheckpointer::new(Recorder, dir, \"model\");","handlingStrategy":"validation","validationCode":"let dir = Path::new(&checkpoint_dir);\nstd::fs::create_dir_all(dir).expect(\"create checkpoint dir\");\nlet probe = dir.join(\".write_probe\");\nstd::fs::write(&probe, b\"ok\").expect(\"checkpoint dir not writable\");\nstd::fs::remove_file(&probe).ok();","typeGuard":"fn checkpoint_dir_writable(dir: &str) -> bool {\n    let p = Path::new(dir).join(\".probe\");\n    std::fs::write(&p, b\"1\").is_ok() && std::fs::remove_file(p).is_ok()\n}","tryCatchPattern":"let result = std::panic::catch_unwind(AssertUnwindSafe(|| checkpointer.checkpoint(actions)));\nif result.is_err() { eprintln!(\"model checkpoint save failed; check disk/permissions\"); }","preventionTips":["Check free disk space periodically during long training runs.","Create and probe the checkpoint directory at startup, before the first save.","Run training as a user with write access to the checkpoint volume.","Save to local disk and sync remotely afterwards for flaky network mounts."],"tags":["rust","panic","io","checkpoint","disk-full","serialization"],"backgroundTag":"checkpoint-io-failure","analyzedSha":"d16f7ba2ed0d41408189384044cc886fb4c8f957","analyzedAt":"2026-09-05T13:19:14.260Z","contentChangedAt":"2026-09-05T13:19:14.260Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}