What causes "out-of-order offset: expected={} actual={}" in clockworklabs/SpacetimeDB?

Ensure exactly one writer per log directory - concurrent access is the most common self-inflicted cause

clockworklabs/SpacetimeDB · critical · io::Error

out-of-order offset: expected={} actual={}

Error message

out-of-order offset: expected={} actual={}

What it means

While traversing a segment to collect Metadata, a commit's first transaction offset does not equal the previous commit's ending offset - a gap or regression within a single segment (InvalidData). Segments must contain strictly contiguous, increasing commits, so a violation means corruption or two writers interleaving appends into the same file.

Source

Thrown at crates/commitlog/src/segment.rs:684

        fn commit_meta<R: io::Read>(
            reader: &mut R,
            sofar: &Metadata,
        ) -> Result<Option<commit::Metadata>, error::SegmentMetadata> {
            commit::Metadata::extract(reader).map_err(|e| {
                if matches!(e.kind(), io::ErrorKind::InvalidData | io::ErrorKind::UnexpectedEof) {
                    error::SegmentMetadata::InvalidCommit {
                        sofar: sofar.clone(),
                        source: e,
                    }
                } else {
                    e.into()
                }
            })
        }
        while let Some(commit) = commit_meta(&mut reader, &sofar)? {
            debug!("commit::{commit:?}");
            if commit.tx_range.start != sofar.tx_range.end {
                return Err(io::Error::new(
                    io::ErrorKind::InvalidData,
                    format!(
                        "out-of-order offset: expected={} actual={}",
                        sofar.tx_range.end, commit.tx_range.start,
                    ),
                )
                .into());
            }
            sofar.tx_range.end = commit.tx_range.end;
            sofar.size_in_bytes += commit.size_in_bytes;
            // TODO: Should it be an error to encounter an epoch going backwards?
            sofar.max_epoch = commit.epoch.max(sofar.max_epoch);
            sofar.max_commit_offset = commit.tx_range.start;
            sofar.max_commit = Some(commit);
        }

        Ok(sofar)
    }

View on GitHub (pinned to 524b4487d9)

Solutions

Ensure exactly one writer per log directory - concurrent access is the most common self-inflicted cause
Quarantine the damaged segment and restore from a replica or backup; do not keep appending to it
If single-writer is provably guaranteed and it still occurs, preserve the file and report it upstream

Defensive patterns

Strategy: try-catch

Type guard

fn is_out_of_order_commit(e: &io::Error) -> bool {
    e.kind() == io::ErrorKind::InvalidData
        && e.to_string().contains("out-of-order offset")
}

Try / catch

match traverse(&segment) {
    Ok(meta) => meta,
    Err(e) if is_out_of_order_commit(&e) => {
        // possible concurrent writer or corruption: stop, quarantine, restore
        std::fs::rename(&segment_path, segment_path.with_extension("quarantine"))?;
        restore_segment_from_replica(&segment_path)?;
        traverse(&segment)?
    }
    Err(e) => return Err(e),
}

Prevention

Enforce single-writer access to each log directory (process supervision, not convention)
Validate segments by full traversal during maintenance windows to catch corruption early
Keep replicas/backups so out-of-order segments can be restored instead of dropped

When it happens

Trigger: Two processes appending to the same segment concurrently (e.g. two instances on one directory); on-disk corruption or bit rot; a segment file spliced or edited by external tooling.

Common situations: Accidental double-start of a service against one data dir; restored or hand-assembled segment files.

Related errors

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/3da08e3401aab9cd. Report an issue: GitHub.