clockworklabs/SpacetimeDB · error · io::Error

mismatched key in offset index file

Error message

mismatched key in offset index file

What it means

During seek_to_offset, the offset index maps a transaction offset to a byte offset, but the commit header decoded at that byte offset carries a different min_tx_offset - index and segment data disagree (InvalidData). The crate re-verifies index entries on read precisely because index writes are flushed asynchronously and can go stale relative to the data.

Source

Thrown at crates/commitlog/src/segment.rs:506

    start_tx_offset: u64,
) -> Result<u64, IndexError> {
    let (index_key, byte_offset) = index_file.key_lookup(start_tx_offset)?;

    // If the index_key is 0, it means the index file is empty, return error without seeking
    if index_key == 0 {
        return Err(IndexError::KeyNotFound);
    }
    debug!("index lookup for key={start_tx_offset}: found key={index_key} at byte-offset={byte_offset}");
    // returned `index_key` should never be greater than `start_tx_offset`
    debug_assert!(index_key <= start_tx_offset);

    // Check if the offset index is pointing to the right commit.
    let hdr = validate_commit_at_byte_offset(&mut segment, byte_offset)?;
    if hdr.min_tx_offset == index_key {
        // Advance the segment Seek if expected commit is found.
        segment.seek(SeekFrom::Start(byte_offset))
    } else {
        Err(io::Error::new(
            io::ErrorKind::InvalidData,
            "mismatched key in offset index file",
        ))
    }
    .map_err(Into::into)
}

/// Try to extract the commit header from the asked position without advancing seek.
/// `IndexFileMut` fsync asynchoronously, which makes it important for reader to verify its entry
fn validate_commit_at_byte_offset<Reader: io::Read + io::Seek>(
    mut reader: &mut Reader,
    byte_offset: u64,
) -> io::Result<commit::Header> {
    let pos = reader.stream_position()?;
    reader.seek(SeekFrom::Start(byte_offset))?;

    let hdr_or_error = StoredCommit::decode(&mut reader).and_then(|maybe_commit| {
        let StoredCommit {

View on GitHub (pinned to 524b4487d9)

Solutions

1. Delete the affected segment's .idx file and reopen: the index is a derived cache and is rebuilt from segment data
2. Keep offset_index_require_segment_fsync = true (the default) so index entries are only added after the segment is fsynced
3. Verify only one process is writing the log directory

fn is_index_key_mismatch(e: &io::Error) -> bool {
    e.kind() == io::ErrorKind::InvalidData
        && e.to_string().contains("mismatched key in offset index file")
}

match reader.seek_to_offset(&index, offset) {
    Ok(_) => { /* proceed */ }
    Err(e) if is_index_key_mismatch(&e) => {
        // the .idx is a derived cache: drop it, reopen, and read via full scan
        drop_the_index_file_for_this_segment()?;
        reopen_and_rescan_from_segment_start(offset)?;
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: A crash between a data write and the asynchronous index flush leaves a stale entry; the segment file was truncated or replaced while its .idx file survived; two writers appending to the same segment concurrently.

Common situations: Recovery after kill -9 with offset_index_require_segment_fsync = false; operators copying or truncating segment files without their .idx.

Related errors

• No valid commit found in index up to key: {candidate_last_ke
• InvalidData
• segment header does not start with magic: expected {:02x?},
• unexpected EOF while validating commit at byte offset {byte_
• out-of-order offset: expected={} actual={}

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/09a8f5d1cad4d928. Report an issue: GitHub.