{"record":{"id":"7d5de1b33b8b1cd5","repo":"clockworklabs/SpacetimeDB","slug":"snapshot-worker-panicked","errorCode":null,"errorMessage":"snapshot worker panicked","messagePattern":"snapshot worker panicked","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/engine/src/snapshot.rs","lineNumber":113,"sourceCode":"            snapshot_created,\n            request_snapshot: request_tx,\n            snapshot_repository,\n        }\n    }\n\n    /// Create a new [SnapshotWorker] on the current Tokio runtime.\n    pub fn new_tokio_current(snapshot_repository: Arc<DynSnapshotRepo>, compression: Compression) -> Self {\n        Self::new(snapshot_repository, compression, Handle::tokio_current())\n    }\n\n    /// Finish the initialization of [Self] by passing a [SnapshotDatabaseState],\n    /// or replace the current [SnapshotDatabaseState] with a new one.\n    ///\n    /// This is called during construction of a [super::relational_db::RelationalDB].\n    pub(crate) fn set_state(&self, state: SnapshotDatabaseState) {\n        self.request_snapshot\n            .unbounded_send(Request::ReplaceState(state))\n            .expect(\"snapshot worker panicked\");\n    }\n\n    /// Get the snapshot repo this worker is operating on.\n    pub fn snapshot_repo(&self) -> Arc<DynSnapshotRepo> {\n        self.snapshot_repository.clone()\n    }\n\n    /// Request a snapshot to be taken.\n    ///\n    /// The snapshot will be taken at some point in the future.\n    /// The request is dropped if the handle is not yet fully initialized.\n    ///\n    /// Panics if the snapshot worker has closed the receive end of its queue(s),\n    /// which is likely due to it having panicked.\n    pub fn request_snapshot(&self) {\n        self.request_snapshot\n            .unbounded_send(Request::TakeSnapshot)\n            .expect(\"snapshot worker panicked\");","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/3653d2ed498fddbd83b7062aa6bf8970e18635ec/crates/engine/src/snapshot.rs#L95-L131","documentation":"Panic when initializing or re-initializing a database's snapshot subsystem. `SnapshotWorker::set_state` sends `Request::ReplaceState(state)` over an unbounded channel and `.expect(\"snapshot worker panicked\")` fires if the snapshot worker task has closed its receiver — meaning the worker panicked or was dropped. Since `set_state` runs during `RelationalDB` construction, this typically aborts database open/startup.","triggerScenarios":"Constructing or updating a `RelationalDB` whose snapshot worker task is no longer alive: the worker panicked earlier (e.g. snapshot write I/O failure, compression error, repository backend error) or its Tokio task was cancelled during runtime teardown and a new DB handle is being created on the same worker.","commonSituations":"Node restart after a snapshot write failure (disk full in snapshot repo, object-store auth errors); tests that build many `RelationalDB` instances on a shared runtime and drop tasks out of order; worker panic caused by a corrupted snapshot repository. The panic surfaces at the NEXT `set_state`/`request_snapshot` call, not at the original failure.","solutions":["Inspect earlier logs for the snapshot worker's original panic (repo write failure, disk space, credentials for remote snapshot storage) and fix that root cause.","Check the snapshot repository location: free space, permissions, or valid object-store credentials, then restart the node.","In test suites, ensure snapshot workers are fully owned by their `RelationalDB` and dropped with it — avoid sharing runtimes across DB instances whose tasks can be cancelled early.","If snapshots are not needed (dev), run the host with snapshotting disabled or clear the snapshot repository so the worker starts clean."],"exampleFix":"// before: DB construction panics because a shared worker already died\nlet db = RelationalDB::open(path).await?; // -> set_state().expect panics\n\n// after: give each DB its own worker lifecycle (and fix the worker's root failure)\nlet worker = SnapshotWorker::new_tokio_current(repo.clone(), compression);\nlet db = RelationalDB::open_with(path, worker).await?;","handlingStrategy":"fallback","validationCode":"// If you manage the worker yourself, probe liveness before set_state\nif worker.is_closed() { /* rebuild the SnapshotWorker instead of letting set_state panic */ }","typeGuard":null,"tryCatchPattern":"let r = std::panic::catch_unwind(AssertUnwindSafe(|| worker.set_state(state)));\nif r.is_err() { /* recreate worker + repository handle, then retry DB open */ }","preventionTips":["Check earlier logs for the worker's root panic (snapshot repo I/O, disk space, credentials).","In tests, tie each SnapshotWorker's lifetime to its RelationalDB; never share across teardown.","Monitor snapshot creation success as a health signal so a dead worker is caught early."],"tags":["rust","spacetimedb","snapshots","worker-panic","channel-closed","database","panic"],"backgroundTag":"background-worker-crashed","analyzedSha":"3653d2ed498fddbd83b7062aa6bf8970e18635ec","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}