clockworklabs/SpacetimeDB · warning · std::io::Error

repo {}: too many empty segments: {}

Error message

repo {}: too many empty segments: {}

What it means

Recovery-time diagnostic from CommittedMeta::extract. While locating the most recent non-empty segment, the opener walks backwards over trailing empty segments (created but never given data, usually after a crash); if it skips more than one, the recovered metadata is still returned but wrapped as CommittedMeta::Prefix with this InvalidData error attached instead of CommittedMeta::Complete. The log stays usable up to the returned metadata - this signals accumulated crash debris, not corruption of the returned data.

Source

Thrown at crates/commitlog/src/commitlog.rs:476

            mut segment_reader,
        }) = open_newest_non_empty_segment(&repo)?
        else {
            return Ok(None);
        };
        let offset_index = repo.get_offset_index(segment_offset).ok();
        match segment::Metadata::extract(segment_offset, &mut segment_reader, offset_index.as_ref()) {
            // Segment is intact.
            Ok(metadata) if empty_segments <= 1 => {
                assert!(
                    !metadata.tx_range.is_empty(),
                    "segment was promised to be non-empty but contains zero transactions"
                );
                Ok(Some(CommittedMeta::Complete { metadata }))
            }
            // Segment is good, but there are too many empty segments.
            Ok(metadata) => Ok(Some(CommittedMeta::Prefix {
                metadata,
                error: io::Error::new(
                    io::ErrorKind::InvalidData,
                    format!("repo {}: too many empty segments: {}", repo, empty_segments),
                ),
            })),
            // Segment is non-empty, but first commit is corrupt.
            Err(error::SegmentMetadata::InvalidCommit { sofar, source }) if sofar.tx_range.is_empty() => {
                Err(io::Error::new(
                    io::ErrorKind::InvalidData,
                    format!(
                        "repo {}: first commit in the most recent segment is corrupt: {}",
                        repo, source
                    ),
                ))
            }
            // Some prefix of the segment is good.
            Err(error::SegmentMetadata::InvalidCommit { sofar, source }) => Ok(Some(CommittedMeta::Prefix {
                metadata: sofar,
                error: source,

View on GitHub (pinned to 9e0d92412f)

Solutions

1. Usually self-healing: once the node commits successfully the empty tail stops growing and the condition disappears on next open.
2. If it recurs, find out why the process dies before its first commit (node logs, OOM, disk full).
3. In a dev environment, wiping and re-initializing the commitlog directory clears the debris.

match committed_meta(repo)? {
    Some(CommittedMeta::Complete { metadata }) => use_metadata(metadata),
    Some(CommittedMeta::Prefix { metadata, error }) => {
        tracing::warn!("log has trailing damage/empties: {error}");
        use_metadata(metadata) // proceed with the good prefix
    }
    None => { /* empty log */ }
}

When it happens

Trigger: A node that crashed and restarted several times in a row, each restart creating a fresh empty segment before dying again; opening the commitlog then counts empty_segments >= 2 and reports the Prefix with this message.

Common situations: Crash loops (OOM killer, failing disk, bad deploy) that repeatedly roll over to a new segment without committing anything; leftover empty segment files from earlier runs in the data directory.

Related errors

• failed to read {} bytes of commit payload: {}
• failed to read checksum: {e}
• refusing to compress mutable segment {head_offset}
• segment {offset} already exists
• segment {offset} does not exist

AI-assisted analysis of clockworklabs/SpacetimeDB@9e0d92412f (2026-08-20). Data as JSON: /api/errors/68cc224558efe5e4. Report an issue: GitHub.