clockworklabs/SpacetimeDB · error · io::Error

unexpected EOF while validating commit at byte offset {byte_

Error message

unexpected EOF while validating commit at byte offset {byte_offset}

What it means

validate_commit_at_byte_offset tried to decode a commit at the byte offset supplied by the index and hit end-of-stream before a complete commit header could be read (UnexpectedEof). The index points past the data end or into the middle of the last, incomplete commit - the classic torn-write signature, possible when index entries become visible before the data is durable.

Source

Thrown at crates/commitlog/src/segment.rs:531

/// Try to extract the commit header from the asked position without advancing seek.
/// `IndexFileMut` fsync asynchoronously, which makes it important for reader to verify its entry
fn validate_commit_at_byte_offset<Reader: io::Read + io::Seek>(
    mut reader: &mut Reader,
    byte_offset: u64,
) -> io::Result<commit::Header> {
    let pos = reader.stream_position()?;
    reader.seek(SeekFrom::Start(byte_offset))?;

    let hdr_or_error = StoredCommit::decode(&mut reader).and_then(|maybe_commit| {
        let StoredCommit {
            min_tx_offset,
            epoch,
            n,
            records,
            ..
        } = maybe_commit.ok_or_else(|| {
            io::Error::new(
                ErrorKind::UnexpectedEof,
                format!("unexpected EOF while validating commit at byte offset {byte_offset}"),
            )
        })?;

        Ok(commit::Header {
            min_tx_offset,
            epoch,
            n,
            len: records.len() as u32,
        })
    });

    // Restore the original position
    reader.seek(SeekFrom::Start(pos))?;

    hdr_or_error
}

View on GitHub (pinned to 524b4487d9)

Solutions

1. Delete the segment's .idx so it is rebuilt from the durable data
2. Keep offset_index_require_segment_fsync = true (default) to close the durability window
3. If it recurs with fsync enabled, suspect the filesystem/device of losing writes and investigate

fn is_index_points_past_eof(e: &io::Error) -> bool {
    e.kind() == io::ErrorKind::UnexpectedEof
        && e.to_string().contains("validating commit at byte offset")
}

match reader.seek_to_offset(&index, offset) {
    Ok(_) => { /* proceed */ }
    Err(e) if is_index_points_past_eof(&e) => {
        // stale index entry after a torn write: rebuild index, rescan from segment start
        drop_the_index_file_for_this_segment()?;
        reopen_and_rescan_from_segment_start(offset)?;
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Process killed mid-commit while offset_index_require_segment_fsync = false (index updated before data was fsynced); a segment truncated while its index retained the old entry; partial page writes on power loss.

Common situations: Crash and fuzz testing; hard power-off on local disks; directories touched by external tooling while the log is in use.

Related errors

• failed to read {} bytes of commit payload: {}
• failed to read checksum: {e}
• InvalidData
• failed to read segment header ({} bytes): {}
• mismatched key in offset index file

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/3dfa7045b9403527. Report an issue: GitHub.