clockworklabs/SpacetimeDB · critical

failed to flush segment upon rotation

Error message

failed to flush segment upon rotation

What it means

Panic in SpacetimeDB's commit log when a segment is rotated. After `commit()` writes transactions, if the current segment exceeds `opts.max_segment_size`, the log flushes, syncs, and starts a new segment; a flush I/O error here hits `.expect("failed to flush segment upon rotation")`. The code intentionally panics because after a partial write the log's internal state is invalid — it cannot know how much data reached disk, and continuing would corrupt the log.

Solutions

1. Free disk space on the volume holding the database/commitlog directory (check `df -h` and inode usage with `df -i`), then restart the SpacetimeDB node — do not try to resume the poisoned in-process log.
2. Verify permissions and open file descriptors on the data directory; ensure no external process prunes segment files.
3. Configure monitoring/disk quotas so the volume cannot silently fill; size `max_segment_size` and retention so rotation pressure is predictable.
4. If the flush failure was transient (e.g. NFS blip), a full restart replays from the last synced segment — inspect logs for torn writes and restore from snapshot if replay fails.

Example fix

// before: letting the disk fill until the commitlog panics on rotation
host.commit(tx).unwrap();

// after: guard free space before committing (ops-level check)
if free_bytes(db_dir) < MIN_FREE_BYTES { return Err(HostError::DiskLow); }
host.commit(tx)?;

// Pre-flight: refuse to commit when the log volume is nearly full
fn can_commit(db_dir: &Path) -> bool {
    fs2::available_space(db_dir).map(|b| b > MIN_FREE_BYTES).unwrap_or(false)
}

// Rust: contain the deliberate poison-panic so the caller can degrade
let r = std::panic::catch_unwind(AssertUnwindSafe(|| log.commit(txns)));
match r { Ok(res) => res?, Err(p) => { /* mark log dead, restart process / failover */ } }

When it happens

Trigger: Calling `CommitLog::commit` repeatedly until the head segment reaches `max_segment_size`, at which point `flush()` returns an I/O error: ENOSPC (disk full), EIO, permission loss on the data directory, or the underlying file being closed/removed underneath the process.

Common situations: Long-running SpacetimeDB host/minion whose database directory fills the disk; container volume limits reached; storage path on a network mount that dropped; running out of inodes; another process deleting segment files. After this panic the commitlog is poisoned (`self.panicked = true`) and every subsequent commit also panics by design.

Related errors

• error reading commit header
• fsync failed
• no space left on device
• segment already exists
• segment does not exist

AI-assisted analysis of clockworklabs/SpacetimeDB@9e0d92412f (2026-08-20). Data as JSON: /api/errors/1155d7f91c77d3cc. Report an issue: GitHub.

Appendix: source

Thrown at crates/commitlog/src/commitlog.rs:317

    ///
    /// - `transactions` exceeds [u16::MAX] elements
    ///
    /// - [Self::flush] or writing to the underlying [Writer] fails
    ///
    ///   This is likely caused by some storage issue. As we cannot tell with
    ///   certainty how much data (if any) has been written, the internal state
    ///   becomes invalid and thus a panic is raised.
    ///
    /// - [Self::sync] panics (called when rotating segments)
    pub fn commit<U: Into<Transaction<T>>>(
        &mut self,
        transactions: impl IntoIterator<Item = U>,
    ) -> io::Result<Option<Committed>> {
        self.panicked = true;
        let writer = &mut self.head;
        let committed = writer.commit(transactions)?;
        if writer.len() >= self.opts.max_segment_size {
            self.flush().expect("failed to flush segment upon rotation");
            self.sync();
            self.start_new_segment()?;
        }
        self.panicked = false;

        Ok(committed)
    }

    pub fn transactions_from<'a, D>(
        &self,
        offset: u64,
        decoder: &'a D,
    ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a + use<'a, D, R, T>
    where
        D: Decoder<Record = T>,
        D::Error: From<error::Traversal>,
        R: 'a,
        T: 'a,

View on GitHub (pinned to 9e0d92412f)