clockworklabs/SpacetimeDB · error · io::Error

expected commit offset {} but encountered {}

Error message

expected commit offset {} but encountered {}

What it means

Contiguity check in append_all_inner: the writer tracks the end of the last written transaction range and requires each incoming commit's min_tx_offset to equal it exactly. A different value means the stream skipped transactions (gap) or replayed already-written ones (overlap), either of which would break the local log's contiguous offset sequence.

Source

Thrown at crates/commitlog/src/stream/writer.rs:327

            // Read the rest of the commit.
            self.commit_buf.body.resize(
                commit_header.len as usize + CHECKSUM_LEN[current_segment.header.checksum_algorithm as usize],
                0,
            );
            stream.read_exact(&mut self.commit_buf.body).await?;
            // Decode the commit and verify its checksum.
            let commit = StoredCommit::decode(self.commit_buf.as_reader())
                .inspect_err(|e| warn!("failed to decode commit: {e}"))?
                .expect("commit decode cannot return `None` because we already decoded the header");

            // Check that the commit offset is what we expect.
            let expected_offset = self
                .last_written_tx_range
                .as_ref()
                .map(|range| range.end)
                .unwrap_or_default();
            if commit.min_tx_offset != expected_offset {
                return Err(io::Error::new(
                    io::ErrorKind::InvalidData,
                    format!(
                        "expected commit offset {} but encountered {}",
                        expected_offset, commit.min_tx_offset
                    ),
                ));
            }
            trace!("received commit {commit:?}");

            // Write the commit and report progress.
            current_segment
                .segment
                .write_all_buf(&mut self.commit_buf.as_buf())
                .await?;
            let written_range = commit.min_tx_offset..(commit.min_tx_offset + commit.n as u64);
            self.last_written_tx_range = Some(written_range.clone());
            progress.range_written(written_range);

View on GitHub (pinned to 524b4487d9)

Solutions

1. Restart the stream from exactly the end of the last written tx range (query repo/writer metadata for the resume point).
2. If local state is disposable, wipe the commitlog and re-bootstrap from offset 0 or a snapshot.
3. Disable any commit filtering between producer and consumer; review leader retention settings.

Example fix

// before: guessing a resume offset
let start = requested_tx_offset; // may not match what was already written

// after: resume from what the local log actually contains
let meta = repo.metadata()?; // last committed tx range
let start = meta.map(|m| m.tx_range.end).unwrap_or(0);

// Before opening the stream, compute the exact resume point from local state:
let start_tx = repo.metadata() // or writer-provided last written range
    .map(|m| m.tx_range.end)
    .unwrap_or(0);
// Ask the producer for commits starting exactly at `start_tx`.

match writer.append_all(stream, progress).await {
    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("expected commit offset") => {
        // Gap or overlap detected: re-bootstrap from a snapshot or restart the stream
        // at the offset reported in the error message.
    }
    r => r,
}

When it happens

Trigger: The producer drops or filters commits between the requested offset and what it actually sends; resuming a stream from an offset that does not match the writer's last_written_tx_range.end; leader-side retention/compaction having removed commits the follower still needs.

Common situations: Follower restart with a stale or hand-picked resume offset; misconfigured filtering on the replication feed; leader retention policy deleting old segments too aggressively.

Related errors

• no current segment, expected segment header
• all-zeroes commit header
• error reading commit header: {e}
• failed to read {} bytes of commit payload: {}
• failed to read checksum: {e}

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/51c2caf395fa9bf2. Report an issue: GitHub.