clockworklabs/SpacetimeDB · critical

Delete for non-existent row when replaying transaction

Error message

Delete for non-existent row when replaying transaction

What it means

Replaying a DELETE operation found no equal row in the rebuilt table: delete_equal_row returned None, meaning the commitlog claims a row existed to delete that the reconstructed committed state does not contain. Replay refuses to silently skip deletes, so it aborts and database open fails.

Source

Thrown at crates/datastore/src/locking_tx_datastore/replay.rs:958

        }

        Ok(())
    }

    fn replay_delete_by_rel(&mut self, table_id: TableId, row: &ProductValue) -> Result<()> {
        // (1) Table dropped? Avoid an error and just ignore the row instead.
        if self.replay_table_dropped.contains(&table_id) {
            return Ok(());
        }

        // Get the table for mutation.
        let (table, blob_store, _, page_pool) = self.get_table_and_blob_store_mut(table_id)?;

        // Delete the row.
        let row_ptr = table
            .delete_equal_row(page_pool, blob_store, row)
            .map_err(TableError::Bflatn)?
            .ok_or_else(|| anyhow!("Delete for non-existent row when replaying transaction"))?;

        if table_id == ST_TABLE_ID {
            let referenced_table_id = row
                .elements
                .get(StTableFields::TableId.col_idx())
                .expect("`st_table` row should conform to `st_table` schema")
                .as_u32()
                .expect("`st_table` row should conform to `st_table` schema");
            if self
                .replay_table_updated
                .remove(&TableId::from(*referenced_table_id))
                .is_some()
            {
                // This delete is part of an update to an `st_table` row,
                // i.e. earlier in this transaction we inserted a new version of the row.
                // That means it's not a dropped table.
            } else {
                // A row was removed from `st_table`, so a table was dropped.

View on GitHub (pinned to 6dee26c6ef)

Solutions

1. Restore from a consistent snapshot and replay only the logs taken after it
2. Ensure every node replays the complete commitlog from the same baseline
3. Use the version that encoded the rows to replay, then upgrade
4. Report suspected corruption with the preserved commitlog

Catch the replay error at open; restore the latest consistent snapshot and replay only segments newer than it; if no snapshot covers the gap, mark the database unrecoverable rather than skipping the delete.

When it happens

Trigger: A truncated or corrupted commitlog that lost the earlier insert of the deleted row; row-encoding (BSATN) changes between versions making row equality fail; replaying only a subset of a database's logs or mixing snapshots and logs from different points in time.

Common situations: Crash mid-commit followed by partial segment replay; copying a data directory while it is being written; upgrading versions that changed row layout without a migration.

Related errors

• Error looking up name for truncated table {table_id:?}
• When replaying `st_column` update: `table_type` should not h
• When replaying `st_column` update: `table_primary_key` shoul
• `CommittedState::replay_columns_to_ignore` should be empty a
• `CommittedState::replay_table_updated` should be empty at th

AI-assisted analysis of clockworklabs/SpacetimeDB@6dee26c6ef (2026-08-20). Data as JSON: /api/errors/349c73491b7805b7. Report an issue: GitHub.