{"record":{"id":"09a8f5d1cad4d928","repo":"clockworklabs/SpacetimeDB","slug":"mismatched-key-in-offset-index-file","errorCode":null,"errorMessage":"mismatched key in offset index file","messagePattern":"mismatched key in offset index file","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"crates/commitlog/src/segment.rs","lineNumber":506,"sourceCode":"    start_tx_offset: u64,\n) -> Result<u64, IndexError> {\n    let (index_key, byte_offset) = index_file.key_lookup(start_tx_offset)?;\n\n    // If the index_key is 0, it means the index file is empty, return error without seeking\n    if index_key == 0 {\n        return Err(IndexError::KeyNotFound);\n    }\n    debug!(\"index lookup for key={start_tx_offset}: found key={index_key} at byte-offset={byte_offset}\");\n    // returned `index_key` should never be greater than `start_tx_offset`\n    debug_assert!(index_key <= start_tx_offset);\n\n    // Check if the offset index is pointing to the right commit.\n    let hdr = validate_commit_at_byte_offset(&mut segment, byte_offset)?;\n    if hdr.min_tx_offset == index_key {\n        // Advance the segment Seek if expected commit is found.\n        segment.seek(SeekFrom::Start(byte_offset))\n    } else {\n        Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            \"mismatched key in offset index file\",\n        ))\n    }\n    .map_err(Into::into)\n}\n\n/// Try to extract the commit header from the asked position without advancing seek.\n/// `IndexFileMut` fsync asynchoronously, which makes it important for reader to verify its entry\nfn validate_commit_at_byte_offset<Reader: io::Read + io::Seek>(\n    mut reader: &mut Reader,\n    byte_offset: u64,\n) -> io::Result<commit::Header> {\n    let pos = reader.stream_position()?;\n    reader.seek(SeekFrom::Start(byte_offset))?;\n\n    let hdr_or_error = StoredCommit::decode(&mut reader).and_then(|maybe_commit| {\n        let StoredCommit {","sourceCodeStart":488,"sourceCodeEnd":524,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/524b4487d949b61a07d4f39c862d1290259dfd20/crates/commitlog/src/segment.rs#L488-L524","documentation":"During seek_to_offset, the offset index maps a transaction offset to a byte offset, but the commit header decoded at that byte offset carries a different min_tx_offset - index and segment data disagree (InvalidData). The crate re-verifies index entries on read precisely because index writes are flushed asynchronously and can go stale relative to the data.","triggerScenarios":"A crash between a data write and the asynchronous index flush leaves a stale entry; the segment file was truncated or replaced while its .idx file survived; two writers appending to the same segment concurrently.","commonSituations":"Recovery after kill -9 with offset_index_require_segment_fsync = false; operators copying or truncating segment files without their .idx.","solutions":["Delete the affected segment's .idx file and reopen: the index is a derived cache and is rebuilt from segment data","Keep offset_index_require_segment_fsync = true (the default) so index entries are only added after the segment is fsynced","Verify only one process is writing the log directory"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"fn is_index_key_mismatch(e: &io::Error) -> bool {\n    e.kind() == io::ErrorKind::InvalidData\n        && e.to_string().contains(\"mismatched key in offset index file\")\n}","tryCatchPattern":"match reader.seek_to_offset(&index, offset) {\n    Ok(_) => { /* proceed */ }\n    Err(e) if is_index_key_mismatch(&e) => {\n        // the .idx is a derived cache: drop it, reopen, and read via full scan\n        drop_the_index_file_for_this_segment()?;\n        reopen_and_rescan_from_segment_start(offset)?;\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Treat .idx files as disposable caches that can always be deleted while the log is closed","Keep offset_index_require_segment_fsync = true so index entries never reference non-durable data","Never truncate or replace segment files while their .idx remains"],"tags":["rust","commitlog","offset-index","index-corruption","invalid-data"],"backgroundTag":"index-corruption","analyzedSha":"524b4487d949b61a07d4f39c862d1290259dfd20","analyzedAt":"2026-08-16T23:58:54.611Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}