clockworklabs/SpacetimeDB · critical · io::Error

failed to fsync file

Error message

failed to fsync file {}: {}

What it means

The final durability step for snapshot files: after opening the file successfully, crates/snapshot calls sync_all() on it. A failure is wrapped (preserving the original kind) as 'failed to fsync file {path}: {source}'. This is the actual flush-to-disk barrier; failure means the file's contents are not guaranteed durable even though writes succeeded.

Solutions

1. Check dmesg/system logs and free space — ENOSPC/EIO at fsync usually means real storage trouble
2. Retry once after clearing space if the kind is StorageFull; otherwise fail the snapshot operation — data is not durable
3. Use a healthy local filesystem for snapshot targets; avoid FUSE/network mounts for durability-critical files
4. Alert on this error: it invalidates the crash-recovery guarantee of the snapshot just written

Example fix

// before
SyncTarget::File(&file).sync_all()?; // EIO: dying disk

// after
if let Err(e) = SyncTarget::File(&file).sync_all() {
    if e.kind() == std::io::ErrorKind::StorageFull {
        // free space then retry once
        cleanup_old_snapshots()?;
        SyncTarget::File(&file).sync_all()?;
    } else {
        return Err(e); // durability violated — surface loudly
    }
}

// Pre-flight headroom so writeback-time ENOSPC is unlikely:
fn disk_headroom(p: &Path) -> bool {
    // platform-specific statvfs; simplify: require > 2x expected snapshot size
    true
}

let mut attempt = 0;
loop {
    match SyncTarget::File(&file).sync_all() {
        Ok(()) => break,
        Err(ref e) if e.kind() == std::io::ErrorKind::StorageFull && attempt == 0 => {
            cleanup_old_snapshots()?; attempt += 1; // free space, retry once
        }
        Err(e) => return Err(e), // durability violated: alert, do not mask
    }
}

When it happens

Trigger: OS-level fsync errors: failing disk (EIO), storage controller issues, ENOSPC discovered at writeback time, interrupted network filesystems, or certain container/overlay filesystems rejecting fsync; extremely long-running sync timing out under load is also seen as errors here.

Common situations: Disk filling up between buffered write and fsync (delayed allocation ENOSPC); dying hardware surfacing at barriers; VM/container hosts with constrained I/O; backup snapshots on external drives that disconnect mid-sync.

Related errors

• failed to fsync directory
• failed to open directory
• failed to open file for fsync
• `close` was called on this `NoDurability` instance
• error syncing data to disk

AI-assisted analysis of clockworklabs/SpacetimeDB@9e0d92412f (2026-08-20). Data as JSON: /api/errors/89aad2c97b0c2d86. Report an issue: GitHub.

Appendix: source

Thrown at crates/snapshot/src/lib.rs:1654

                .map_err(|e| io::Error::new(e.kind(), format!("failed to fsync directory {}: {}", path.display(), e))),
            Self::File(path) => {
                File::options()
                    .read(true)
                    // Windows needs the file to be writable for `sync_all` to work.
                    // Set all the open options explicitly, just for visibility.
                    .write(true)
                    .truncate(false)
                    .create(false)
                    .append(false)
                    .open(path)
                    .map_err(|e| {
                        io::Error::new(
                            e.kind(),
                            format!("failed to open file {} for fsync: {}", path.display(), e),
                        )
                    })?
                    .sync_all()
                    .map_err(|e| io::Error::new(e.kind(), format!("failed to fsync file {}: {}", path.display(), e)))
            }
        }
    }
}

#[cfg(test)]
mod tests {
    use std::fs::OpenOptions;

    use tempfile::tempdir;

    use super::*;

    #[test]
    fn listing_ignores_if_snapshot_file_is_missing() -> anyhow::Result<()> {
        let tmp = tempdir()?;

        let root = SnapshotsPath::from_path_unchecked(tmp.path());

View on GitHub (pinned to 9e0d92412f)