clockworklabs/SpacetimeDB · critical

failed to flush segment upon rotation

Error message

failed to flush segment upon rotation

What it means

SpacetimeDB's commit log stores transactions in size-bounded segment files. When `Commitlog::commit` grows the head segment past `opts.max_segment_size`, it must flush (write buffered data and fsync) before starting a new segment; this `expect` fires when that flush returns an I/O error. Because a failed flush leaves unknown how much data reached disk, the log poisons its state (note the `panicked` flag) and aborts rather than continue with invalid state.

Source

Thrown at crates/commitlog/src/commitlog.rs:317

    ///
    /// - `transactions` exceeds [u16::MAX] elements
    ///
    /// - [Self::flush] or writing to the underlying [Writer] fails
    ///
    ///   This is likely caused by some storage issue. As we cannot tell with
    ///   certainty how much data (if any) has been written, the internal state
    ///   becomes invalid and thus a panic is raised.
    ///
    /// - [Self::sync] panics (called when rotating segments)
    pub fn commit<U: Into<Transaction<T>>>(
        &mut self,
        transactions: impl IntoIterator<Item = U>,
    ) -> io::Result<Option<Committed>> {
        self.panicked = true;
        let writer = &mut self.head;
        let committed = writer.commit(transactions)?;
        if writer.len() >= self.opts.max_segment_size {
            self.flush().expect("failed to flush segment upon rotation");
            self.sync();
            self.start_new_segment()?;
        }
        self.panicked = false;

        Ok(committed)
    }

    pub fn transactions_from<'a, D>(
        &self,
        offset: u64,
        decoder: &'a D,
    ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a + use<'a, D, R, T>
    where
        D: Decoder<Record = T>,
        D::Error: From<error::Traversal>,
        R: 'a,
        T: 'a,

View on GitHub (pinned to 524b4487d9)

Solutions

1. Check free space and quotas on the commitlog data directory (df -h, container disk limits); ENOSPC is the most common cause — free space or expand the volume.
2. Inspect dmesg/journalctl for I/O errors on the underlying device and run smartctl/fsck if the disk is unhealthy.
3. Verify the data directory and segment files are writable by the spacetimedb process user (permissions, read-only mounts).
4. Restart the node after remediation; do not reuse in-memory commitlog state after this panic — it is intentionally poisoned.
5. Review max_segment_size and retention settings so the volume has headroom.

Example fix

// before: rotation flush failure aborts the process inside commit()
log.commit(txs)?;

// after: catch the panic at the call site and surface it as an error
let committed = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| log.commit(txs)))
    .map_err(|_| io::Error::new(io::ErrorKind::Other, "commitlog flush failed during rotation"))??;

// Before committing, check the segment directory has room for a full rotation
// (nix crate): let vfs = nix::sys::statvfs::statvfs(data_dir)?;
// let free = vfs.blocks_available() as u64 * vfs.block_size() as u64;
// assert!(free > max_segment_size + headroom);

let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| log.commit(txs))); match r { Ok(res) => { /* process Ok(res) */ } Err(_) => { /* log dir + disk state; restart node — state is poisoned */ } }

When it happens

Trigger: Calling `Commitlog::commit(transactions)` repeatedly until `writer.len() >= opts.max_segment_size` triggers rotation; the panic happens only if the subsequent `self.flush()` errors — ENOSPC (disk full), EIO (failing device), EACCES/EBADF on the segment file, or an fsync error surfaced by the OS.

Common situations: Host disk filling up because segment retention keeps too much data; running the node on a volume with an exhausted quota; failing disk or filesystem corruption; a container with a small tmpfs or read-only mount for the data directory.

Related errors

• fsync failed
• error syncing data to disk
• failed to decode commit
• no current segment, expected segment header
• failed to fsync file {}: {}

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/1155d7f91c77d3cc. Report an issue: GitHub.