clockworklabs/SpacetimeDB · error · std::io::Error

failed to read bytes of commit payload

Error message

failed to read {} bytes of commit payload: {}

What it means

Emitted by Commit::decode after a commit header promises hdr.len bytes of record payload and read_exact for exactly those bytes fails. The most common cause is a torn write: the process died after the header reached disk but before all payload bytes did, so the log ends mid-commit. The original error kind is preserved and the expected byte count is included in the message.

Solutions

1. Let normal recovery run: SpacetimeDB truncates the log back to the last complete commit on open, discarding the torn tail.
2. If recovery does not clear it, truncate the affected segment to its last valid commit boundary (or remove the damaged newest segment) and restart.
3. Fix the root cause: stop force-killing the process; verify fsync behavior, disk health and free space.

match reader.next_commit() {
    Ok(Some(commit)) => commits.push(commit),
    Ok(None) => {} // end of log
    Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => {
        // torn tail from a crash: stop at the last complete commit
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Decoding a commit whose payload extends past the end of the segment file - typically after a crash or power loss between writing the commit header and its records, or after external truncation of a segment.

Common situations: Hard-killed nodes (kill -9, OOM, power cut) replaying their commitlog on restart; segment files truncated by an outside process; disk full at the moment of the write.

Related errors

• failed to read checksum
• all-zeroes commit header
• error reading commit header
• failed to flush segment upon rotation
• Failed to truncate offset index

AI-assisted analysis of clockworklabs/SpacetimeDB@6dee26c6ef (2026-08-20). Data as JSON: /api/errors/b25e9c42d3ade0d6. Report an issue: GitHub.

Appendix: source

Thrown at crates/commitlog/src/commit.rs:314

    /// [`ChecksumMismatch`] is returned.
    pub fn decode<R: Read>(reader: R) -> io::Result<Option<Self>> {
        Self::decode_internal(reader, DEFAULT_LOG_FORMAT_VERSION)
    }

    pub(crate) fn decode_internal<R: Read>(reader: R, log_format_version: u8) -> io::Result<Option<Self>> {
        let mut reader = Crc32cReader::new(reader);

        let v = if log_format_version == 0 {
            Version::V0
        } else {
            Version::V1
        };
        let Some(hdr) = Header::decode_internal(&mut reader, v)? else {
            return Ok(None);
        };
        let mut records = vec![0; hdr.len as usize];
        reader.read_exact(&mut records).map_err(|e| {
            io::Error::new(
                e.kind(),
                format!("failed to read {} bytes of commit payload: {}", hdr.len, e),
            )
        })?;

        let chk = reader.crc32c();
        let crc = decode_u32(reader.into_inner())
            .map_err(|e| io::Error::new(e.kind(), format!("failed to read checksum: {e}")))?;

        if chk != crc {
            return Err(invalid_data(ChecksumMismatch));
        }

        Ok(Some(Self {
            min_tx_offset: hdr.min_tx_offset,
            epoch: hdr.epoch,
            n: hdr.n,
            records,

View on GitHub (pinned to 6dee26c6ef)