clockworklabs/SpacetimeDB · error · io::Error

mismatch key in index offset file: expected=

Error message

mismatch key in index offset file: expected={} actual={}

What it means

validate_commit_at_offset successfully decodes a commit header at the index's byte offset, but its tx_range.start differs from the tx_offset key stored in the index. The entry maps a transaction offset to the wrong byte position, returning io::ErrorKind::InvalidData with expected/actual values. This is the detectable form of a stale offset index after a crash or partial restore.

Solutions

1. Rebuild the offset index by deleting index files and reopening the log
2. Restore segment and index files only as matched pairs from the same snapshot
3. Handle the error by falling back to a linear scan for the requested offset
4. Log expected vs actual offsets to confirm which direction the index is stale

fn is_index_offset_mismatch(err: &io::Error) -> bool {
    err.kind() == io::ErrorKind::InvalidData && err.to_string().contains("mismatch key in index offset file")
}

match validate_commit_at_offset(&mut reader, tx_offset, byte_offset) {
    Ok(meta) => Ok(meta),
    Err(e) if is_index_offset_mismatch(&e) => {
        // entry maps to the wrong byte position: scan from the previous valid entry
        scan_for_tx(&mut reader, tx_offset)
    }
    Err(e) => Err(e),
}

When it happens

Trigger: Index entry not yet updated after commits were rewritten at new byte positions (async index fsync plus crash); segment and index from different backup generations; concurrent external modification of segment files.

Common situations: Unclean shutdown between segment write and index flush; restoring a segment file but keeping the old index; copying logs between nodes with mismatched index state.

Related errors

• mismatched key in offset index file
• failed to decode commit
• No valid commit found in index up to key
• all-zeroes commit header
• Failed to truncate offset index

AI-assisted analysis of clockworklabs/SpacetimeDB@3653d2ed49 (2026-08-20). Data as JSON: /api/errors/8f92bf7b0c744d6a. Report an issue: GitHub.

Appendix: source

Thrown at crates/commitlog/src/segment.rs:775

        ))
    }

    /// Validates and decodes a commit at `byte_offset` in the segment.
    ///
    /// # Returns
    /// * `Ok(commit::Metadata)` - If a valid commit is found with matching transaction offset
    /// * `Err` - If commit can't be decoded or has mismatched transaction offset
    fn validate_commit_at_offset<R: io::Read + io::Seek>(
        reader: &mut R,
        tx_offset: TxOffset,
        byte_offset: u64,
    ) -> io::Result<commit::Metadata> {
        reader.seek(SeekFrom::Start(byte_offset))?;
        let commit = commit::Metadata::extract(reader)?
            .ok_or_else(|| io::Error::new(ErrorKind::InvalidData, "failed to decode commit"))?;

        if commit.tx_range.start != tx_offset {
            return Err(io::Error::new(
                ErrorKind::InvalidData,
                format!(
                    "mismatch key in index offset file: expected={} actual={}",
                    tx_offset, commit.tx_range.start
                ),
            ));
        }

        Ok(commit)
    }
}

#[cfg(test)]
mod tests {
    use itertools::Itertools;
    use pretty_assertions::assert_matches;
    use spacetimedb_paths::server::CommitLogDir;
    use tempfile::tempdir;

View on GitHub (pinned to 3653d2ed49)