{"record":{"id":"3da08e3401aab9cd","repo":"clockworklabs/SpacetimeDB","slug":"out-of-order-offset-expected-actual","errorCode":null,"errorMessage":"out-of-order offset: expected={} actual={}","messagePattern":"out-of-order offset: expected=(.+?) actual=(.+?)","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"critical","filePath":"crates/commitlog/src/segment.rs","lineNumber":684,"sourceCode":"        fn commit_meta<R: io::Read>(\n            reader: &mut R,\n            sofar: &Metadata,\n        ) -> Result<Option<commit::Metadata>, error::SegmentMetadata> {\n            commit::Metadata::extract(reader).map_err(|e| {\n                if matches!(e.kind(), io::ErrorKind::InvalidData | io::ErrorKind::UnexpectedEof) {\n                    error::SegmentMetadata::InvalidCommit {\n                        sofar: sofar.clone(),\n                        source: e,\n                    }\n                } else {\n                    e.into()\n                }\n            })\n        }\n        while let Some(commit) = commit_meta(&mut reader, &sofar)? {\n            debug!(\"commit::{commit:?}\");\n            if commit.tx_range.start != sofar.tx_range.end {\n                return Err(io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    format!(\n                        \"out-of-order offset: expected={} actual={}\",\n                        sofar.tx_range.end, commit.tx_range.start,\n                    ),\n                )\n                .into());\n            }\n            sofar.tx_range.end = commit.tx_range.end;\n            sofar.size_in_bytes += commit.size_in_bytes;\n            // TODO: Should it be an error to encounter an epoch going backwards?\n            sofar.max_epoch = commit.epoch.max(sofar.max_epoch);\n            sofar.max_commit_offset = commit.tx_range.start;\n            sofar.max_commit = Some(commit);\n        }\n\n        Ok(sofar)\n    }","sourceCodeStart":666,"sourceCodeEnd":702,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/524b4487d949b61a07d4f39c862d1290259dfd20/crates/commitlog/src/segment.rs#L666-L702","documentation":"While traversing a segment to collect Metadata, a commit's first transaction offset does not equal the previous commit's ending offset - a gap or regression within a single segment (InvalidData). Segments must contain strictly contiguous, increasing commits, so a violation means corruption or two writers interleaving appends into the same file.","triggerScenarios":"Two processes appending to the same segment concurrently (e.g. two instances on one directory); on-disk corruption or bit rot; a segment file spliced or edited by external tooling.","commonSituations":"Accidental double-start of a service against one data dir; restored or hand-assembled segment files.","solutions":["Ensure exactly one writer per log directory - concurrent access is the most common self-inflicted cause","Quarantine the damaged segment and restore from a replica or backup; do not keep appending to it","If single-writer is provably guaranteed and it still occurs, preserve the file and report it upstream"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_out_of_order_commit(e: &io::Error) -> bool {\n    e.kind() == io::ErrorKind::InvalidData\n        && e.to_string().contains(\"out-of-order offset\")\n}","tryCatchPattern":"match traverse(&segment) {\n    Ok(meta) => meta,\n    Err(e) if is_out_of_order_commit(&e) => {\n        // possible concurrent writer or corruption: stop, quarantine, restore\n        std::fs::rename(&segment_path, segment_path.with_extension(\"quarantine\"))?;\n        restore_segment_from_replica(&segment_path)?;\n        traverse(&segment)?\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Enforce single-writer access to each log directory (process supervision, not convention)","Validate segments by full traversal during maintenance windows to catch corruption early","Keep replicas/backups so out-of-order segments can be restored instead of dropped"],"tags":["rust","commitlog","segment","corruption","out-of-order","invalid-data"],"backgroundTag":"out-of-order-sequence","analyzedSha":"524b4487d949b61a07d4f39c862d1290259dfd20","analyzedAt":"2026-08-16T23:58:54.611Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}