clockworklabs/SpacetimeDB · error · io::Error

mismatched key in offset index file

Error message

mismatched key in offset index file

What it means

After an offset-index lookup for a transaction offset, the reader seeks to the recorded byte offset and re-reads the commit header to verify it. If the commit's min_tx_offset does not equal the index key, the index disagrees with the segment data and io::ErrorKind::InvalidData 'mismatched key in offset index file' is returned. Verification exists because IndexFileMut fsyncs asynchronously, so after a crash an entry can point at a stale position.

Solutions

1. Delete the offset index files so they are rebuilt from segment data on next open
2. Always copy segment and its index together when archiving/restoring
3. Fall back to the backwards-scan path (as done around segment.rs:745) or a full segment scan when the direct seek fails
4. Report the (key, byte_offset, found min_tx_offset) triple for diagnosis

fn is_index_key_mismatch(err: &io::Error) -> bool {
    err.kind() == io::ErrorKind::InvalidData && err.to_string().contains("mismatched key in offset index file")
}

match segment.seek_to_tx(tx_offset).await {
    Ok(pos) => Ok(pos),
    Err(e) if is_index_key_mismatch(&e) => {
        // index is untrustworthy: fall back to a full sequential scan of the segment
        scan_segment_for_tx(&mut segment, tx_offset).await
    }
    Err(e) => Err(e),
}

When it happens

Trigger: Crash between writing commit data to the segment and flushing the offset index; index file truncated or partially written; copying a segment file into the repo without its matching index (or vice versa); seeking an old tx offset whose index entry predates a segment rewrite.

Common situations: Unclean shutdowns; restoring backups that mix segment generations; manual file manipulation in the log directory.

Related errors

• mismatch key in index offset file: expected=
• failed to decode commit
• No valid commit found in index up to key
• all-zeroes commit header
• Failed to truncate offset index

AI-assisted analysis of clockworklabs/SpacetimeDB@3653d2ed49 (2026-08-20). Data as JSON: /api/errors/09a8f5d1cad4d928. Report an issue: GitHub.

Appendix: source

Thrown at crates/commitlog/src/segment.rs:515

    start_tx_offset: u64,
) -> Result<u64, IndexError> {
    let (index_key, byte_offset) = index_file.key_lookup(start_tx_offset)?;

    // If the index_key is 0, it means the index file is empty, return error without seeking
    if index_key == 0 {
        return Err(IndexError::KeyNotFound);
    }
    debug!("index lookup for key={start_tx_offset}: found key={index_key} at byte-offset={byte_offset}");
    // returned `index_key` should never be greater than `start_tx_offset`
    debug_assert!(index_key <= start_tx_offset);

    // Check if the offset index is pointing to the right commit.
    let hdr = validate_commit_at_byte_offset(&mut segment, byte_offset)?;
    if hdr.min_tx_offset == index_key {
        // Advance the segment Seek if expected commit is found.
        segment.seek(SeekFrom::Start(byte_offset))
    } else {
        Err(io::Error::new(
            io::ErrorKind::InvalidData,
            "mismatched key in offset index file",
        ))
    }
    .map_err(Into::into)
}

/// Try to extract the commit header from the asked position without advancing seek.
/// `IndexFileMut` fsync asynchoronously, which makes it important for reader to verify its entry
fn validate_commit_at_byte_offset<Reader: io::Read + io::Seek>(
    mut reader: &mut Reader,
    byte_offset: u64,
) -> io::Result<commit::Header> {
    let pos = reader.stream_position()?;
    reader.seek(SeekFrom::Start(byte_offset))?;

    let hdr_or_error = StoredCommit::decode(&mut reader).and_then(|maybe_commit| {
        let StoredCommit {

View on GitHub (pinned to 3653d2ed49)