clockworklabs/SpacetimeDB · critical

fsync failed

Error message

fsync failed

What it means

`AsyncFsync` is SpacetimeDB's internal trait for durability on async files; the impl for `tokio::fs::File` calls `sync_data().await.expect("fsync failed")`, so any error from the OS fsync aborts the task. An fsync error means the OS could not guarantee data reached stable storage — typically ENOSPC, EIO, or a vanished/failed backend device.

Source

Thrown at crates/commitlog/src/stream/common.rs:48

    fn open_segment_reader_async(
        &self,
        offset: u64,
    ) -> impl Future<Output = io::Result<Self::AsyncSegmentReader>> + Send;
}

pub trait AsyncFsync {
    fn fsync(&self) -> impl Future<Output = ()> + Send;
}

impl<T: AsyncWrite + AsyncFsync + Send + Sync> AsyncFsync for tokio::io::BufWriter<T> {
    async fn fsync(&self) {
        self.get_ref().fsync().await
    }
}

impl AsyncFsync for tokio::fs::File {
    async fn fsync(&self) {
        self.sync_data().await.expect("fsync failed")
    }
}

pub trait AsyncLen: AsyncSeek + Unpin + Send {
    fn segment_len(&mut self) -> impl Future<Output = io::Result<u64>> + Send
    where
        Self: Sized,
    {
        async { spacetimedb_fs_utils::compression::segment_len(self).await }
    }
}

impl<T: AsyncWrite + AsyncLen + Send> AsyncLen for tokio::io::BufWriter<T> {
    async fn segment_len(&mut self) -> io::Result<u64> {
        self.get_mut().segment_len().await
    }
}

View on GitHub (pinned to 524b4487d9)

Solutions

1. Free or expand space on the affected filesystem (df -h / container limits) — ENOSPC is the most frequent trigger.
2. Check kernel logs (dmesg, journalctl -k) for EIO or device errors; test with smartctl and replace faulty hardware.
3. Move the database data directory off NFS/FUSE onto local ext4/xfs storage.
4. Restart the process after fixing the underlying cause; already-open files may be in an inconsistent state.

Example fix

// before: the trait impl panics on any fsync error
async fn fsync(&self) { self.sync_data().await.expect("fsync failed") }

// after (if you control the call site): propagate instead of panicking
async fn fsync_checked(file: &tokio::fs::File) -> io::Result<()> {
    file.sync_data().await
}

// Preflight free space before heavy writes (nix crate statvfs on the data dir);
// abort writes when available bytes fall below one segment + headroom.

When it happens

Trigger: Any code path that fsyncs a commitlog segment or other async-written file on a full, failing, or detached filesystem: quota exhausted mid-write, dm-crypt/NFS backend returning EIO, or the block device being removed underneath the process.

Common situations: Containers with small ephemeral volumes; data directories placed on NFS/SMB or FUSE filesystems with unreliable fsync semantics; failing SSDs or HW RAID members; ENOSPC after large burst writes.

Related errors

• failed to flush segment upon rotation
• error syncing data to disk
• failed to fsync file {}: {}
• never types are not yet supported in C# output
• unions not supported

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/c4ca8bc77b0f35af. Report an issue: GitHub.