clockworklabs/SpacetimeDB · error · io::Error

failed to decode commit

Error message

failed to decode commit

What it means

Thrown by validate_commit_at_offset (crates/commitlog/src/segment.rs) while cross-checking a commitlog segment against its offset index. The code seeks to the byte offset recorded in the index and asks commit::Metadata::extract to decode a commit there; the read succeeded but extract returned None, meaning the bytes at that position do not form a valid commit record (typically a zeroed or padded region). This signals segment corruption or a segment/index mismatch, not a transient I/O failure.

Source

Thrown at crates/commitlog/src/segment.rs:763

        Err(io::Error::new(
            ErrorKind::InvalidData,
            format!("No valid commit found in index up to key: {candidate_last_key}"),
        ))
    }

    /// Validates and decodes a commit at `byte_offset` in the segment.
    ///
    /// # Returns
    /// * `Ok(commit::Metadata)` - If a valid commit is found with matching transaction offset
    /// * `Err` - If commit can't be decoded or has mismatched transaction offset
    fn validate_commit_at_offset<R: io::Read + io::Seek>(
        reader: &mut R,
        tx_offset: TxOffset,
        byte_offset: u64,
    ) -> io::Result<commit::Metadata> {
        reader.seek(SeekFrom::Start(byte_offset))?;
        let commit = commit::Metadata::extract(reader)?
            .ok_or_else(|| io::Error::new(ErrorKind::InvalidData, "failed to decode commit"))?;

        if commit.tx_range.start != tx_offset {
            return Err(io::Error::new(
                ErrorKind::InvalidData,
                format!(
                    "mismatch key in index offset file: expected={} actual={}",
                    tx_offset, commit.tx_range.start
                ),
            ));
        }

        Ok(commit)
    }
}

#[cfg(test)]
mod tests {
    use itertools::Itertools;

View on GitHub (pinned to 524b4487d9)

Solutions

1. Delete the segment's offset index files so they are rebuilt by rescanning the segment, then re-run verification.
2. If the segment tail itself is corrupt, truncate or remove the affected segment and re-replicate it from a leader or snapshot.
3. Check disk health (dmesg, smartctl) and free space to rule out hardware-induced zero fills.
4. Restore the entire commitlog directory as a unit from a consistent backup instead of individual files.

use std::fs;
use std::io;

// Cheap pre-check before validating indexed commits: the segment must
// physically contain at least a commit header at `byte_offset`.
fn segment_covers_offset(segment_path: &str, byte_offset: u64) -> io::Result<bool> {
    let len = fs::metadata(segment_path)?.len();
    Ok(len > byte_offset + MIN_COMMIT_RECORD_LEN) // e.g. commit header + checksum size
}

match validate_result {
    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("failed to decode commit") => {
        // Segment/index disagree: rebuild the offset index or drop and re-replicate
        // the segment. Do NOT retry the same read - corruption is deterministic.
    }
    other => other,
}

When it happens

Trigger: Running segment verification/repair APIs that walk every (tx_offset -> byte_offset) entry of a segment's offset index and validate the commit at each entry. Fires when byte_offset lands in zero-filled preallocated (fallocate) space beyond the last written commit, in a region torn by a crash mid-write, or in data from a different generation of the segment file.

Common situations: Hard kill or power loss during commitlog writes leaving a partially written tail; a full or failing disk producing zero-filled regions; mixing a recreated segment file with an index from a previous run; partially copying or restoring a commitlog repository by hand.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• InvalidData
• out-of-order offset: expected={} actual={}
• mismatch key in index offset file: expected={} actual={}
• InvalidInput
• AlreadyExists

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/62e5f3173e43b594. Report an issue: GitHub.