clockworklabs/SpacetimeDB · critical

stored reset program changed

Error message

stored reset program changed

What it means

After persisting the new program bytes with `program_store.put`, `reset_database` asserts the store's returned content hash equals the hash computed from the in-memory program. A mismatch means the stored bytes differ from what was validated with `check_module_validity_with_environment`, i.e. the program storage backend returned a different hash than expected — an internal integrity violation, so the reset is aborted before swapping replicas.

Solutions

1. Verify the program store's hashing algorithm and serialization match how `program.hash` was computed (same hash function, same byte slice).
2. Recompute the hash from the exact bytes passed to `put` and compare before calling put to isolate where the mismatch originates.
3. Check for concurrent mutation of `program.bytes` between hashing, validity checking, and storage; clone/own the bytes early.
4. Reproduce against the default program store; if only a custom store fails, fix or roll back the custom store.

Example fix

// before
let stored = self.program_store.put(&program.bytes).await?;
anyhow::ensure!(stored == program.hash, "stored reset program changed");
// after (verify before persisting)
let recomputed = Hash::from_bytes(&Sha256::digest(&program.bytes));
anyhow::ensure!(recomputed == program.hash, "program hash mismatch before storing");
let stored = self.program_store.put(&program.bytes).await?;
anyhow::ensure!(stored == program.hash, "stored reset program changed");

use sha2::{Digest, Sha256};
let recomputed: [u8; 32] = Sha256::digest(&program.bytes).into();
assert_eq!(&recomputed, program.hash.as_slice(), "program hash drifted before storage");

fn hash_matches(program: &Program, stored: &Hash) -> bool { stored == &program.hash }

When it happens

Trigger: `program_store.put(&program.bytes)` returns a hash that differs from the precomputed `program.hash`, e.g. a buggy/misconfigured content-addressed store, hash-algorithm mismatch between the program builder and the store, or corrupted/mutated program bytes between hashing and storage.

Common situations: Custom or sharded program stores with a hashing bug; upgrading SpacetimeDB so the store's hashing no longer matches; running with a storage backend (S3/local) that altered or truncated the bytes; concurrency bugs where bytes were modified during an async put.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• allocation size overflow
• error reading commit header
• failed to flush segment upon rotation
• fsync failed
• invalid seek to a negative or overflowing position

AI-assisted analysis of clockworklabs/SpacetimeDB@eddf9f5014 (2026-09-20). Data as JSON: /api/errors/8a5abdb9e2d3dbd5. Report an issue: GitHub.

Appendix: source

Thrown at crates/standalone/src/lib.rs:484

                Program::from_bytes(host_type.into(), &bytes[..])
            }
            None => {
                // A reset without an artifact retains the currently committed
                // module, not the initial program or its old environment values.
                let module = self.leader(database.id).await?.module().await?;
                module
                    .relational_db()
                    .program()?
                    .context("database is not initialized")?
            }
        };
        database.host_type = HostType::from(program.kind);
        database.initial_program = program.hash;
        self.host_controller
            .check_module_validity_with_environment(database.clone(), program.clone(), environment.clone())
            .await?;
        let stored = self.program_store.put(&program.bytes).await?;
        anyhow::ensure!(stored == program.hash, "stored reset program changed");
        let previous_replicas = self.control_db.get_replicas_by_database(database.id)?;
        for replica in &previous_replicas {
            self.on_delete_replica(replica.id).await?;
        }
        let (_, replica) = self.control_db.upsert_database_with_environment(
            database,
            Some(&previous),
            environment,
            &previous_replicas,
        )?;
        self.on_insert_replica(&replica).await?;
        Ok(())
    }

    async fn add_energy(&self, identity: &Identity, amount: EnergyQuanta) -> anyhow::Result<()> {
        let balance = self
            .control_db
            .get_energy_balance(identity)?

View on GitHub (pinned to eddf9f5014)