tracel-ai/burn · error

Error when deleting the state: {err}

Error message

Error when deleting the state: {err}

What it means

Sentinel guard in the training checkpointing background thread: when a checkpoint save/delete I/O operation fails and no interrupter (graceful shutdown channel) is available, the async checkpoint handler panics, aborting the checkpoint worker thread because training cannot safely continue without reliable checkpoint persistence.

Source

Thrown at crates/burn-train/src/checkpoint/async_checkpoint.rs:54

                                    "Error when sending response through callback channel: {err}"
                                )
                            },
                            |int| int.stop(Some(&err.to_string())),
                        )
                    });
                }
                Message::Save(epoch, state, interrupter) => {
                    self.checkpointer.save(epoch, state).unwrap_or_else(|err| {
                        interrupter.map_or_else(
                            || panic!("Error when saving the state: {err}"),
                            |int| int.stop(Some(&err.to_string())),
                        )
                    });
                }
                Message::Delete(epoch, interrupter) => {
                    self.checkpointer.delete(epoch).unwrap_or_else(|err| {
                        interrupter.map_or_else(
                            || panic!("Error when deleting the state: {err}"),
                            |int| int.stop(Some(&err.to_string())),
                        )
                    });
                }

                Message::End => {
                    return;
                }
            };
        }
    }
}

/// Async checkpointer.
pub struct AsyncCheckpointer<R> {
    sender: mpsc::SyncSender<Message<R>>,
    handler: Option<std::thread::JoinHandle<()>>,
    interrupter: Option<Interrupter>,

View on GitHub (pinned to d16f7ba2ed)

Solutions

1. Check disk space and filesystem permissions on the checkpoint directory
2. Verify the checkpoint path exists and is writable by the training process
3. Inspect the wrapped `err` for the underlying I/O cause (e.g., device full, permission denied)
4. If the interrupter was available, prefer the graceful-stop path instead of reaching this panic

When it happens

Trigger: Thrown at crates/burn-train/src/checkpoint/async_checkpoint.rs:54 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of tracel-ai/burn@d16f7ba2ed (2026-09-05). Data as JSON: /api/errors/521e841dfe1ff923. Report an issue: GitHub.