{"record":{"id":"62e5f3173e43b594","repo":"clockworklabs/SpacetimeDB","slug":"failed-to-decode-commit","errorCode":null,"errorMessage":"failed to decode commit","messagePattern":"failed to decode commit","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"crates/commitlog/src/segment.rs","lineNumber":763,"sourceCode":"        Err(io::Error::new(\n            ErrorKind::InvalidData,\n            format!(\"No valid commit found in index up to key: {candidate_last_key}\"),\n        ))\n    }\n\n    /// Validates and decodes a commit at `byte_offset` in the segment.\n    ///\n    /// # Returns\n    /// * `Ok(commit::Metadata)` - If a valid commit is found with matching transaction offset\n    /// * `Err` - If commit can't be decoded or has mismatched transaction offset\n    fn validate_commit_at_offset<R: io::Read + io::Seek>(\n        reader: &mut R,\n        tx_offset: TxOffset,\n        byte_offset: u64,\n    ) -> io::Result<commit::Metadata> {\n        reader.seek(SeekFrom::Start(byte_offset))?;\n        let commit = commit::Metadata::extract(reader)?\n            .ok_or_else(|| io::Error::new(ErrorKind::InvalidData, \"failed to decode commit\"))?;\n\n        if commit.tx_range.start != tx_offset {\n            return Err(io::Error::new(\n                ErrorKind::InvalidData,\n                format!(\n                    \"mismatch key in index offset file: expected={} actual={}\",\n                    tx_offset, commit.tx_range.start\n                ),\n            ));\n        }\n\n        Ok(commit)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use itertools::Itertools;","sourceCodeStart":745,"sourceCodeEnd":781,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/524b4487d949b61a07d4f39c862d1290259dfd20/crates/commitlog/src/segment.rs#L745-L781","documentation":"Thrown by validate_commit_at_offset (crates/commitlog/src/segment.rs) while cross-checking a commitlog segment against its offset index. The code seeks to the byte offset recorded in the index and asks commit::Metadata::extract to decode a commit there; the read succeeded but extract returned None, meaning the bytes at that position do not form a valid commit record (typically a zeroed or padded region). This signals segment corruption or a segment/index mismatch, not a transient I/O failure.","triggerScenarios":"Running segment verification/repair APIs that walk every (tx_offset -> byte_offset) entry of a segment's offset index and validate the commit at each entry. Fires when byte_offset lands in zero-filled preallocated (fallocate) space beyond the last written commit, in a region torn by a crash mid-write, or in data from a different generation of the segment file.","commonSituations":"Hard kill or power loss during commitlog writes leaving a partially written tail; a full or failing disk producing zero-filled regions; mixing a recreated segment file with an index from a previous run; partially copying or restoring a commitlog repository by hand.","solutions":["Delete the segment's offset index files so they are rebuilt by rescanning the segment, then re-run verification.","If the segment tail itself is corrupt, truncate or remove the affected segment and re-replicate it from a leader or snapshot.","Check disk health (dmesg, smartctl) and free space to rule out hardware-induced zero fills.","Restore the entire commitlog directory as a unit from a consistent backup instead of individual files."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"use std::fs;\nuse std::io;\n\n// Cheap pre-check before validating indexed commits: the segment must\n// physically contain at least a commit header at `byte_offset`.\nfn segment_covers_offset(segment_path: &str, byte_offset: u64) -> io::Result<bool> {\n    let len = fs::metadata(segment_path)?.len();\n    Ok(len > byte_offset + MIN_COMMIT_RECORD_LEN) // e.g. commit header + checksum size\n}","typeGuard":null,"tryCatchPattern":"match validate_result {\n    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains(\"failed to decode commit\") => {\n        // Segment/index disagree: rebuild the offset index or drop and re-replicate\n        // the segment. Do NOT retry the same read - corruption is deterministic.\n    }\n    other => other,\n}","preventionTips":["Shut nodes down cleanly so commitlog tails are never torn mid-commit.","Never mix index files from one data directory into another; always move whole repos.","Keep the commitlog on a local filesystem with health monitoring and free-space alerts.","Run segment verification periodically so corruption is caught early, not during recovery."],"tags":["commitlog","corruption","segment","rust","spacetimedb"],"backgroundTag":"commitlog-segment-corruption","analyzedSha":"524b4487d949b61a07d4f39c862d1290259dfd20","analyzedAt":"2026-08-16T23:58:54.611Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}