clockworklabs/SpacetimeDB · critical

fsync failed

Error message

fsync failed

What it means

Panic inside SpacetimeDB's async commitlog stream layer: the `AsyncFsync` implementation for `tokio::fs::File` calls `sync_data().await.expect("fsync failed")`. Any I/O error from the OS `fsync`/`fdatasync` call — hardware error, dropped network storage, file already closed — becomes a panic instead of a `Result`, taking down the task that awaited it.

Solutions

1. Check disk health immediately (`dmesg | grep -i 'i/o error'`, SMART status) — an fsync failure usually means real data risk; restore the database from snapshot after fixing hardware.
2. Ensure the data directory is on reliable local storage, not a flaky network mount, and that quotas/limits leave headroom for sync.
3. Restart the node after the storage fault clears; treat the last partially synced segment as suspect (replay from the previous good snapshot).
4. If you control the code, prefer returning `io::Result` from custom `AsyncFsync` impls instead of panicking, so上层 durability code can handle failures gracefully.

Example fix

// before: panic on fsync error inside the trait impl
impl AsyncFsync for tokio::fs::File {
    async fn fsync(&self) { self.sync_data().await.expect("fsync failed") }
}

// after (for custom storage): surface the error
impl AsyncFsync for MyFile {
    async fn fsync(&self) {
        if let Err(e) = self.sync_data().await { /* log and propagate */ }
    }
}

// Pre-check before relying on sync: ensure the fd is valid and space remains
if available_space(data_dir)? < MIN_FREE_BYTES { return Err(io::Error::new(ErrorKind::StorageFull, "low disk")); }

let res = future::FutureExt::catch_unwind(std::pin::pin!(file.fsync()));
if res.await.is_err() { /* storage fault: stop writer, escalate, restart process */ }

When it happens

Trigger: Any code path that syncs a commitlog segment file: segment rotation, durability flushes, or explicit sync of the log writer. `sync_data()` returns Err on EIO/EBADF/ENOSPC (e.g. failing disk, file closed early, thin-provisioned volume out of space), and the `expect` converts it to a panic inside the async task.

Common situations: Failing or ejecting disk; container overlay filesystem errors; NFS/SMB disconnect mid-sync; disk quota hit during fdatasync; test harnesses closing files early. Because the panic occurs in a `Send + Sync` impl shared by writer tasks, it usually cascades into the 'log worker panicked' / commitlog poisoning panics (errors 603/605).

Related errors

• error syncing data to disk
• failed to flush segment upon rotation
• log worker panicked
• Duration overflows i64 microseconds
• Duration since Unix epoch overflows i64 microseconds

AI-assisted analysis of clockworklabs/SpacetimeDB@9e0d92412f (2026-08-20). Data as JSON: /api/errors/c4ca8bc77b0f35af. Report an issue: GitHub.

Appendix: source

Thrown at crates/commitlog/src/stream/common.rs:51

    fn open_segment_reader_async(
        &self,
        offset: u64,
    ) -> impl Future<Output = io::Result<Self::AsyncSegmentReader>> + Send;
}

pub trait AsyncFsync {
    fn fsync(&self) -> impl Future<Output = ()> + Send;
}

impl<T: AsyncWrite + AsyncFsync + Send + Sync> AsyncFsync for tokio::io::BufWriter<T> {
    async fn fsync(&self) {
        self.get_ref().fsync().await
    }
}

impl AsyncFsync for tokio::fs::File {
    async fn fsync(&self) {
        self.sync_data().await.expect("fsync failed")
    }
}

pub trait AsyncLen: AsyncSeek + Unpin + Send {
    fn segment_len(&mut self) -> impl Future<Output = io::Result<u64>> + Send
    where
        Self: Sized,
    {
        async { spacetimedb_fs_utils::compression::segment_len(self).await }
    }
}

impl<T: AsyncWrite + AsyncLen + Send> AsyncLen for tokio::io::BufWriter<T> {
    async fn segment_len(&mut self) -> io::Result<u64> {
        self.get_mut().segment_len().await
    }
}

View on GitHub (pinned to 9e0d92412f)