clockworklabs/SpacetimeDB · error · io::Error

failed to fsync directory

Error message

failed to fsync directory {}: {}

What it means

After successfully opening a snapshot directory for fsync, crates/snapshot calls sync_all() on the directory File. If the OS-level fsync on the directory fails, the underlying io::Error kind is preserved and wrapped with 'failed to fsync directory {path}: {source}'. Directory fsync is what makes renames/creations inside it durable; a failure means the directory-entry changes may not survive a crash.

Solutions

1. Match the wrapped ErrorKind: for InvalidInput/Unsupported from exotic filesystems, treat directory fsync as best-effort and log-and-continue if your durability model permits
2. Move the snapshot store to a POSIX-compliant local filesystem (ext4/xfs) when directory durability is required
3. Investigate hardware/kernel logs (dmesg) if EIO — the storage itself is unhealthy
4. Do not silently swallow: at minimum record which directory failed so recovery can re-verify

Example fix

// before
SyncTarget::Dir(&dir).sync_all()?; // EIO/EINVAL from FUSE mount

// after (best-effort directory fsync on exotic filesystems)
if let Err(e) = SyncTarget::Dir(&dir).sync_all() {
    tracing::warn!(?dir, %e, "directory fsync failed; durability of dir entries not guaranteed");
}

// Nothing to validate at the API level; check the mount supports it once at startup:
let probe = File::open(&snapshot_dir)?.sync_all();
if probe.is_err() { tracing::warn!("directory fsync unsupported on this mount"); }

if let Err(e) = SyncTarget::Dir(&dir).sync_all() {
    match e.kind() {
        std::io::ErrorKind::Unsupported | std::io::ErrorKind::InvalidInput => {
            tracing::warn!(?dir, %e, "dir fsync unsupported; durability best-effort");
        }
        _ => return Err(e),
    }
}

When it happens

Trigger: Hardware/filesystem-level fsync errors (I/O error, failing disk); filesystems that do not support fsync on directory file descriptors (some FUSE/NFS configurations); interrupted underlying storage; ResourceExhausted conditions on some kernels under heavy sync load.

Common situations: Running on network or exotic filesystems (NFS, certain FUSE mounts, some container storage drivers) where directory fsync returns EINVAL/EIO; failing disks surfacing errors at barrier operations; test environments on overlayfs.

Related errors

• failed to fsync file
• failed to open directory
• failed to open file for fsync
• `close` was called on this `NoDurability` instance
• error syncing data to disk

AI-assisted analysis of clockworklabs/SpacetimeDB@9e0d92412f (2026-08-20). Data as JSON: /api/errors/6525fb3e767f8ce1. Report an issue: GitHub.

Appendix: source

Thrown at crates/snapshot/src/lib.rs:1636

    /// On *nix systems, both a file and its enclosing directory should be
    /// `fsync`ed to make the file durable.
    ///
    /// On Windows, only the file needs to be synced, and it's even an error to
    /// sync a directory. Passing in [Self::Dir] is thus a no-op on Windows.
    fn sync_all(&self) -> io::Result<()> {
        match self {
            #[cfg(target_os = "windows")]
            Self::Dir(path) => Ok(()),
            #[cfg(not(target_os = "windows"))]
            Self::Dir(path) => File::open(path)
                .map_err(|e| {
                    io::Error::new(
                        e.kind(),
                        format!("failed to open directory {} for fsync: {}", path.display(), e),
                    )
                })?
                .sync_all()
                .map_err(|e| io::Error::new(e.kind(), format!("failed to fsync directory {}: {}", path.display(), e))),
            Self::File(path) => {
                File::options()
                    .read(true)
                    // Windows needs the file to be writable for `sync_all` to work.
                    // Set all the open options explicitly, just for visibility.
                    .write(true)
                    .truncate(false)
                    .create(false)
                    .append(false)
                    .open(path)
                    .map_err(|e| {
                        io::Error::new(
                            e.kind(),
                            format!("failed to open file {} for fsync: {}", path.display(), e),
                        )
                    })?
                    .sync_all()
                    .map_err(|e| io::Error::new(e.kind(), format!("failed to fsync file {}: {}", path.display(), e)))

View on GitHub (pinned to 9e0d92412f)