clockworklabs/SpacetimeDB · error · io::Error

repo {}: segment {} already exists and is non-empty: {}

Error message

repo {}: segment {} already exists and is non-empty: {}

What it means

When the mirrored stream presents a segment header, StreamWriter tries create_segment. On io::ErrorKind::AlreadyExists it opens the existing segment and only overwrites when its length is at most segment::Header::LEN (a stub). An existing segment with real data is refused with AlreadyExists 'segment N already exists and is non-empty', protecting the destination from being overwritten.

Source

Thrown at crates/commitlog/src/stream/writer.rs:466

    let segment_offset = last_written_tx_range
        .as_ref()
        .map(|range| range.end)
        .unwrap_or_default();
    let mut segment = loop {
        match repo.create_segment(segment_offset, header) {
            Ok(segment) => break segment,
            Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {
                trace!("segment already exists");
                let mut s = repo.open_segment_writer(segment_offset)?;
                let len = s.segment_len()?;
                trace!("segment len: {len}");
                if len <= segment::Header::LEN as _ {
                    trace!("overwriting existing segment");
                    repo.remove_segment(segment_offset)?;
                    continue;
                }

                return Err(io::Error::new(
                    e.kind(),
                    format!(
                        "repo {}: segment {} already exists and is non-empty: {}",
                        repo, segment_offset, e
                    ),
                ));
            }
            Err(e) => return Err(e),
        }
    };
    fallocate(&mut segment, &commitlog_options)?;

    let index_writer = repo
        .create_offset_index(segment_offset, commitlog_options.offset_index_len())
        .inspect_err(|e| warn!("unable to create offset index segment={segment_offset} err={e:?}"))
        .map(|index| OffsetIndexWriter::new(index, commitlog_options))
        .ok();

View on GitHub (pinned to 9e0d92412f)

Solutions

1. Replay into a fresh, empty repo/directory when starting a stream from the beginning
2. If resuming, align the stream start with the destination's existing content instead of replaying already-written segments
3. Verify the existing segment's data is disposable (it matches the stream's earlier content) before removing it
4. Ensure only one StreamWriter writes to a repo at a time

// Ensure the destination is empty (or absent) before replaying from the start
let offsets = repo.existing_offsets()?;
if !offsets.is_empty() {
    return Err(io::Error::new(
        io::ErrorKind::AlreadyExists,
        format!("destination repo not empty: {offsets:?}"),
    ));
}
let writer = StreamWriter::create(repo, opts, OnTrailingData::Trim)?;

fn is_nonempty_segment_exists(err: &io::Error) -> bool {
    err.kind() == io::ErrorKind::AlreadyExists && err.to_string().contains("already exists and is non-empty")
}

match writer.append_all(&mut stream, &mut progress).await {
    Ok(done) => Ok(done),
    Err(e) if is_nonempty_segment_exists(&e) => {
        // stop; a human must decide whether existing data is disposable
        Err(e)
    }
    Err(e) => Err(e),
}

When it happens

Trigger: Replaying a mirror stream into a repo directory that already contains data for those segment offsets: restarting a restore from an earlier point without clearing the target; pointing a second mirror job at the same repo; wrong destination directory that already holds a commitlog.

Common situations: Re-running restore jobs over partially populated directories; misconfigured destination paths in automation; concurrent mirror writers targeting one repo.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

• no current segment, expected segment header
• all-zeroes commit header
• expected commit offset {} but encountered {}
• repo {}: segment {} already exists and is non-empty
• failed to read {} bytes of commit payload: {}

AI-assisted analysis of clockworklabs/SpacetimeDB@9e0d92412f (2026-08-20). Data as JSON: /api/errors/9f72307fb29bebbd. Report an issue: GitHub.