clockworklabs/SpacetimeDB · error · anyhow::Error

unable to lock database {} for update

Error message

unable to lock database {} for update

What it means

update_database takes the per-replica write lock before swapping in a new program version. Acquisition is capped by a 5-second timeout; if a concurrent operation (running reducers, another update, host teardown) still holds the lock when the timeout elapses, the module update fails with this error naming the database identity.

Source

Thrown at crates/core/src/host/host_controller.rs:567

    pub async fn update_module_host(
        &self,
        database: Database,
        host_type: HostType,
        replica_id: u64,
        program_bytes: Box<[u8]>,
        policy: MigrationPolicy,
    ) -> anyhow::Result<UpdateDatabaseResult> {
        let program = Program::from_bytes(host_type.into(), program_bytes);
        trace!(
            "update module host {}/{}: genesis={} update-to={}",
            database.database_identity,
            replica_id,
            database.initial_program,
            program.hash
        );

        let Ok(mut guard) = self.acquire_write_lock(replica_id).await else {
            bail!("unable to lock database {} for update", database.database_identity);
        };

        // `HostController::clone` is fast,
        // as all of its fields are either `Copy` or wrapped in `Arc`.
        let this = self.clone();
        let database_identity = database.database_identity;

        // `try_init_host` is not cancel safe, as it will spawn other async tasks
        // which hold a filesystem lock past when `try_init_host` returns or is cancelled.
        // This means that, if `try_init_host` is cancelled, subsequent calls will fail.
        //
        // The rest of this future is also not cancel safe, as it will `Option::take` out of the guard
        // at the start of the block and then store back into it at the end.
        //
        // This is problematic because Axum will cancel its handler tasks if the client disconnects,
        // and this method is called from Axum handlers, e.g. for the publish route.
        // `tokio::spawn` a task to update the contents of `guard`,
        // so that it will run to completion even if the caller goes away.

View on GitHub (pinned to fdd647dfac)

Solutions

1. Retry the publish once the concurrent operation drains
2. Check server logs for the lock holder / deadlock symptoms (this timeout was introduced to debug exactly that)
3. Restart the node if the lock appears stuck with no live operation
4. Schedule updates during low traffic or gate them with a deploy lock

# shell: retry the update after the lock holder drains
for i in 1 2 3; do
  spacetime publish my-db --project-path . && exit 0
  echo "attempt $i: replica lock busy during update" >&2
  sleep 10
done
exit 1

When it happens

Trigger: Publishing an update to a database whose replica write lock is held >5s by another operation -- e.g. an in-flight reducer call, a concurrent publish, or a previous update still initializing; or a stuck holder never releasing the lock.

Common situations: CI deploying while players are actively triggering reducers on a busy database; two deploy pipelines racing; retrying an update immediately after a cancelled one whose tasks still hold the lock.

Related errors

• unable to lock database {} for initialization
• unable to lock database {} for migration planning
• database `{addr}` not yet initialized
• Missing type name for ${typeBuilder.constructor.name ?? 'Typ
• unexpected update error: {e}

AI-assisted analysis of clockworklabs/SpacetimeDB@fdd647dfac (2026-08-20). Data as JSON: /api/errors/d2b8dc1b3c0eaeb1. Report an issue: GitHub.