clockworklabs/SpacetimeDB · critical

snapshot worker panicked

Error message

snapshot worker panicked

What it means

Panic when initializing or re-initializing a database's snapshot subsystem. `SnapshotWorker::set_state` sends `Request::ReplaceState(state)` over an unbounded channel and `.expect("snapshot worker panicked")` fires if the snapshot worker task has closed its receiver — meaning the worker panicked or was dropped. Since `set_state` runs during `RelationalDB` construction, this typically aborts database open/startup.

Source

Thrown at crates/engine/src/snapshot.rs:113

            snapshot_created,
            request_snapshot: request_tx,
            snapshot_repository,
        }
    }

    /// Create a new [SnapshotWorker] on the current Tokio runtime.
    pub fn new_tokio_current(snapshot_repository: Arc<DynSnapshotRepo>, compression: Compression) -> Self {
        Self::new(snapshot_repository, compression, Handle::tokio_current())
    }

    /// Finish the initialization of [Self] by passing a [SnapshotDatabaseState],
    /// or replace the current [SnapshotDatabaseState] with a new one.
    ///
    /// This is called during construction of a [super::relational_db::RelationalDB].
    pub(crate) fn set_state(&self, state: SnapshotDatabaseState) {
        self.request_snapshot
            .unbounded_send(Request::ReplaceState(state))
            .expect("snapshot worker panicked");
    }

    /// Get the snapshot repo this worker is operating on.
    pub fn snapshot_repo(&self) -> Arc<DynSnapshotRepo> {
        self.snapshot_repository.clone()
    }

    /// Request a snapshot to be taken.
    ///
    /// The snapshot will be taken at some point in the future.
    /// The request is dropped if the handle is not yet fully initialized.
    ///
    /// Panics if the snapshot worker has closed the receive end of its queue(s),
    /// which is likely due to it having panicked.
    pub fn request_snapshot(&self) {
        self.request_snapshot
            .unbounded_send(Request::TakeSnapshot)
            .expect("snapshot worker panicked");

View on GitHub (pinned to 3653d2ed49)

Solutions

1. Inspect earlier logs for the snapshot worker's original panic (repo write failure, disk space, credentials for remote snapshot storage) and fix that root cause.
2. Check the snapshot repository location: free space, permissions, or valid object-store credentials, then restart the node.
3. In test suites, ensure snapshot workers are fully owned by their `RelationalDB` and dropped with it — avoid sharing runtimes across DB instances whose tasks can be cancelled early.
4. If snapshots are not needed (dev), run the host with snapshotting disabled or clear the snapshot repository so the worker starts clean.

Example fix

// before: DB construction panics because a shared worker already died
let db = RelationalDB::open(path).await?; // -> set_state().expect panics

// after: give each DB its own worker lifecycle (and fix the worker's root failure)
let worker = SnapshotWorker::new_tokio_current(repo.clone(), compression);
let db = RelationalDB::open_with(path, worker).await?;

// If you manage the worker yourself, probe liveness before set_state
if worker.is_closed() { /* rebuild the SnapshotWorker instead of letting set_state panic */ }

let r = std::panic::catch_unwind(AssertUnwindSafe(|| worker.set_state(state)));
if r.is_err() { /* recreate worker + repository handle, then retry DB open */ }

When it happens

Trigger: Constructing or updating a `RelationalDB` whose snapshot worker task is no longer alive: the worker panicked earlier (e.g. snapshot write I/O failure, compression error, repository backend error) or its Tokio task was cancelled during runtime teardown and a new DB handle is being created on the same worker.

Common situations: Node restart after a snapshot write failure (disk full in snapshot repo, object-store auth errors); tests that build many `RelationalDB` instances on a shared runtime and drop tasks out of order; worker panic caused by a corrupted snapshot repository. The panic surfaces at the NEXT `set_state`/`request_snapshot` call, not at the original failure.

Related errors

• log worker panicked
• table should exist in the database for AddConstraint
• length didn't fit in `u32`
• unable to decode args
• unable to encode view return value

AI-assisted analysis of clockworklabs/SpacetimeDB@3653d2ed49 (2026-08-20). Data as JSON: /api/errors/7d5de1b33b8b1cd5. Report an issue: GitHub.