{"record":{"id":"3da08e3401aab9cd","repo":"clockworklabs/SpacetimeDB","slug":"out-of-order-offset-expected-actual","errorCode":null,"errorMessage":"out-of-order offset: expected={} actual={}","messagePattern":"out-of-order offset: expected=(.+?) actual=(.+?)","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"critical","filePath":"crates/commitlog/src/segment.rs","lineNumber":693,"sourceCode":"        fn commit_meta<R: io::Read + SegmentPos>(\n            reader: &mut R,\n            sofar: &Metadata,\n        ) -> Result<Option<commit::Metadata>, error::SegmentMetadata> {\n            commit::Metadata::extract(reader).map_err(|e| {\n                if matches!(e.kind(), io::ErrorKind::InvalidData | io::ErrorKind::UnexpectedEof) {\n                    error::SegmentMetadata::InvalidCommit {\n                        sofar: sofar.clone(),\n                        source: e,\n                    }\n                } else {\n                    e.into()\n                }\n            })\n        }\n        while let Some(commit) = commit_meta(&mut reader, &sofar)? {\n            debug!(\"commit::{commit:?}\");\n            if commit.tx_range.start != sofar.tx_range.end {\n                return Err(io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    format!(\n                        \"out-of-order offset: expected={} actual={}\",\n                        sofar.tx_range.end, commit.tx_range.start,\n                    ),\n                )\n                .into());\n            }\n            sofar.tx_range.end = commit.tx_range.end;\n            sofar.size_in_bytes += commit.size_in_bytes;\n            // TODO: Should it be an error to encounter an epoch going backwards?\n            sofar.max_epoch = commit.epoch.max(sofar.max_epoch);\n            sofar.max_commit_offset = commit.tx_range.start;\n            sofar.max_commit = Some(commit);\n        }\n\n        Ok(sofar)\n    }","sourceCodeStart":675,"sourceCodeEnd":711,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/3653d2ed498fddbd83b7062aa6bf8970e18635ec/crates/commitlog/src/segment.rs#L675-L711","documentation":"While traversing a segment to build its Metadata (during resume or StreamWriter::create), every commit's tx_range.start must equal the previous commit's tx_range.end. A gap between consecutive commits returns io::ErrorKind::InvalidData 'out-of-order offset: expected=X actual=Y', wrapped as SegmentMetadata::InvalidCommit when a partial Metadata was already collected. The segment cannot be resumed as-is.","triggerScenarios":"A segment containing commits with gaps: torn writes that still decode successfully, manually spliced/concatenated segment files, or data from two different segments interleaved; replaying a segment produced by a buggy producer.","commonSituations":"Post-crash corruption that happens to checksum out; hand-repaired log directories; version-skewed producers writing incompatible commit layouts that misparse.","solutions":["Archive and remove the affected segment and re-stream its data from a replica/upstream","If the gap is in the trailing region, trim trailing invalid data on open (the OnTrailingData hook of StreamWriter::create exists for this)","Capture the full traversal Metadata and the expected/actual offsets for a bug report if no corruption is evident","Verify the segment file was not modified externally (compare with an archived copy)"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_out_of_order_segment(err: &io::Error) -> bool {\n    err.kind() == io::ErrorKind::InvalidData && err.to_string().contains(\"out-of-order offset\")\n}","tryCatchPattern":"match segment::Metadata::extract(offset, &mut segment, index.as_ref()) {\n    Ok(meta) => Ok(meta),\n    Err(e) if is_out_of_order_segment(&e) => {\n        // segment is unresumable: archive it and re-stream its range from upstream\n        archive_segment(repo, offset)?;\n        restream_from_upstream(repo, offset).await\n    }\n    Err(e) => Err(e),\n}","preventionTips":["Never splice or concatenate segment files by hand","Trim trailing invalid data on open so torn tails cannot look like gaps between commits","Keep an upstream source of truth for every segment so quarantining is lossless"],"tags":["rust","commitlog","contiguity","corruption","invalid-data"],"backgroundTag":"sequence-number-gap","analyzedSha":"3653d2ed498fddbd83b7062aa6bf8970e18635ec","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}