clockworklabs/SpacetimeDB · error · io::Error

expected commit offset

Error message

expected commit offset {} but encountered {}

What it means

StreamWriter enforces commit-offset contiguity across the whole stream: each commit's min_tx_offset must equal the end of the last written tx range (0 when nothing was written yet). A gap or duplicate returns io::ErrorKind::InvalidData 'expected commit offset X but encountered Y', identifying exactly where the stream and destination diverged.

Solutions

1. Query the destination's actual position (StreamWriter::create_and_metadata returns the last segment's Metadata; use its tx_range.end) and request the source stream from exactly that offset
2. On mismatch, do not skip forward blindly: restart the mirror from a verified common offset into a clean state
3. Verify the transport (retry buffering, dedup) is not dropping or duplicating stream chunks
4. Log the expected/actual pair to identify whether data was lost (gap) or repeated (duplicate)

// Align the source stream start with the destination before writing
let (_, meta) = StreamWriter::create_and_metadata(repo.clone(), opts.clone(), OnTrailingData::Trim)?;
let resume_from = meta.as_ref().map(|m| m.tx_range.end).unwrap_or(0);
// request the source to stream commits starting exactly at `resume_from`
let stream = source.stream_from_tx(resume_from).await?;

fn is_stream_offset_gap(err: &io::Error) -> bool {
    err.kind() == io::ErrorKind::InvalidData && err.to_string().contains("expected commit offset")
}

match writer.append_all(&mut stream, &mut progress).await {
    Ok(done) => Ok(done),
    Err(e) if is_stream_offset_gap(&e) => {
        // destination and source diverged: tear down and restart the mirror aligned
        restart_mirror_aligned(&repo, &source).await
    }
    Err(e) => Err(e),
}

When it happens

Trigger: Starting the source stream at an offset that does not continue the destination's last written range; dropped or duplicated data in the network transport; the producer restarting from a different point mid-stream; resuming a mirror job with a stale source offset.

Common situations: Reconnecting replication with a wrong 'resume from' offset; source and destination out of sync after a partial failure; replaying an old stream snippet over newer data.

Related errors

• all-zeroes commit header
• invalid transaction offset
• no current segment, expected segment header
• out-of-order offset: expected=
• repo : segment already exists and is non-empty

AI-assisted analysis of clockworklabs/SpacetimeDB@9e0d92412f (2026-08-20). Data as JSON: /api/errors/51c2caf395fa9bf2. Report an issue: GitHub.

Appendix: source

Thrown at crates/commitlog/src/stream/writer.rs:327

            // Read the rest of the commit.
            self.commit_buf.body.resize(
                commit_header.len as usize + CHECKSUM_LEN[current_segment.header.checksum_algorithm as usize],
                0,
            );
            stream.read_exact(&mut self.commit_buf.body).await?;
            // Decode the commit and verify its checksum.
            let commit = StoredCommit::decode(&mut self.commit_buf.as_reader(bytes_written))
                .inspect_err(|e| warn!("failed to decode commit: {e}"))?
                .expect("commit decode cannot return `None` because we already decoded the header");

            // Check that the commit offset is what we expect.
            let expected_offset = self
                .last_written_tx_range
                .as_ref()
                .map(|range| range.end)
                .unwrap_or_default();
            if commit.min_tx_offset != expected_offset {
                return Err(io::Error::new(
                    io::ErrorKind::InvalidData,
                    format!(
                        "expected commit offset {} but encountered {}",
                        expected_offset, commit.min_tx_offset
                    ),
                ));
            }
            trace!("received commit {commit:?}");

            // Write the commit and report progress.
            current_segment
                .segment
                .write_all_buf(&mut self.commit_buf.as_buf())
                .await?;
            let written_range = commit.min_tx_offset..(commit.min_tx_offset + commit.n as u64);
            self.last_written_tx_range = Some(written_range.clone());
            progress.range_written(written_range);

View on GitHub (pinned to 9e0d92412f)