clockworklabs/SpacetimeDB · critical · io::Error

out-of-order offset: expected=

Error message

out-of-order offset: expected={} actual={}

What it means

While traversing a segment to build its Metadata (during resume or StreamWriter::create), every commit's tx_range.start must equal the previous commit's tx_range.end. A gap between consecutive commits returns io::ErrorKind::InvalidData 'out-of-order offset: expected=X actual=Y', wrapped as SegmentMetadata::InvalidCommit when a partial Metadata was already collected. The segment cannot be resumed as-is.

Solutions

1. Archive and remove the affected segment and re-stream its data from a replica/upstream
2. If the gap is in the trailing region, trim trailing invalid data on open (the OnTrailingData hook of StreamWriter::create exists for this)
3. Capture the full traversal Metadata and the expected/actual offsets for a bug report if no corruption is evident
4. Verify the segment file was not modified externally (compare with an archived copy)

fn is_out_of_order_segment(err: &io::Error) -> bool {
    err.kind() == io::ErrorKind::InvalidData && err.to_string().contains("out-of-order offset")
}

match segment::Metadata::extract(offset, &mut segment, index.as_ref()) {
    Ok(meta) => Ok(meta),
    Err(e) if is_out_of_order_segment(&e) => {
        // segment is unresumable: archive it and re-stream its range from upstream
        archive_segment(repo, offset)?;
        restream_from_upstream(repo, offset).await
    }
    Err(e) => Err(e),
}

When it happens

Trigger: A segment containing commits with gaps: torn writes that still decode successfully, manually spliced/concatenated segment files, or data from two different segments interleaved; replaying a segment produced by a buggy producer.

Common situations: Post-crash corruption that happens to checksum out; hand-repaired log directories; version-skewed producers writing incompatible commit layouts that misparse.

Related errors

• no valid commits in segment
• segment header does not start with magic: expected
• all-zeroes commit header
• Delete for non-existent row when replaying transaction
• Error looking up name for truncated table

AI-assisted analysis of clockworklabs/SpacetimeDB@3653d2ed49 (2026-08-20). Data as JSON: /api/errors/3da08e3401aab9cd. Report an issue: GitHub.

Appendix: source

Thrown at crates/commitlog/src/segment.rs:693

        fn commit_meta<R: io::Read + SegmentPos>(
            reader: &mut R,
            sofar: &Metadata,
        ) -> Result<Option<commit::Metadata>, error::SegmentMetadata> {
            commit::Metadata::extract(reader).map_err(|e| {
                if matches!(e.kind(), io::ErrorKind::InvalidData | io::ErrorKind::UnexpectedEof) {
                    error::SegmentMetadata::InvalidCommit {
                        sofar: sofar.clone(),
                        source: e,
                    }
                } else {
                    e.into()
                }
            })
        }
        while let Some(commit) = commit_meta(&mut reader, &sofar)? {
            debug!("commit::{commit:?}");
            if commit.tx_range.start != sofar.tx_range.end {
                return Err(io::Error::new(
                    io::ErrorKind::InvalidData,
                    format!(
                        "out-of-order offset: expected={} actual={}",
                        sofar.tx_range.end, commit.tx_range.start,
                    ),
                )
                .into());
            }
            sofar.tx_range.end = commit.tx_range.end;
            sofar.size_in_bytes += commit.size_in_bytes;
            // TODO: Should it be an error to encounter an epoch going backwards?
            sofar.max_epoch = commit.epoch.max(sofar.max_epoch);
            sofar.max_commit_offset = commit.tx_range.start;
            sofar.max_commit = Some(commit);
        }

        Ok(sofar)
    }

View on GitHub (pinned to 3653d2ed49)