{"record":{"id":"89aad2c97b0c2d86","repo":"clockworklabs/SpacetimeDB","slug":"failed-to-fsync-file","errorCode":null,"errorMessage":"failed to fsync file {}: {}","messagePattern":"failed to fsync file (.+?): (.+?)","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"critical","filePath":"crates/snapshot/src/lib.rs","lineNumber":1654,"sourceCode":"                .map_err(|e| io::Error::new(e.kind(), format!(\"failed to fsync directory {}: {}\", path.display(), e))),\n            Self::File(path) => {\n                File::options()\n                    .read(true)\n                    // Windows needs the file to be writable for `sync_all` to work.\n                    // Set all the open options explicitly, just for visibility.\n                    .write(true)\n                    .truncate(false)\n                    .create(false)\n                    .append(false)\n                    .open(path)\n                    .map_err(|e| {\n                        io::Error::new(\n                            e.kind(),\n                            format!(\"failed to open file {} for fsync: {}\", path.display(), e),\n                        )\n                    })?\n                    .sync_all()\n                    .map_err(|e| io::Error::new(e.kind(), format!(\"failed to fsync file {}: {}\", path.display(), e)))\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::fs::OpenOptions;\n\n    use tempfile::tempdir;\n\n    use super::*;\n\n    #[test]\n    fn listing_ignores_if_snapshot_file_is_missing() -> anyhow::Result<()> {\n        let tmp = tempdir()?;\n\n        let root = SnapshotsPath::from_path_unchecked(tmp.path());","sourceCodeStart":1636,"sourceCodeEnd":1672,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/9e0d92412ff2248f401a8ad12d535f2b5ac30912/crates/snapshot/src/lib.rs#L1636-L1672","documentation":"The final durability step for snapshot files: after opening the file successfully, crates/snapshot calls sync_all() on it. A failure is wrapped (preserving the original kind) as 'failed to fsync file {path}: {source}'. This is the actual flush-to-disk barrier; failure means the file's contents are not guaranteed durable even though writes succeeded.","triggerScenarios":"OS-level fsync errors: failing disk (EIO), storage controller issues, ENOSPC discovered at writeback time, interrupted network filesystems, or certain container/overlay filesystems rejecting fsync; extremely long-running sync timing out under load is also seen as errors here.","commonSituations":"Disk filling up between buffered write and fsync (delayed allocation ENOSPC); dying hardware surfacing at barriers; VM/container hosts with constrained I/O; backup snapshots on external drives that disconnect mid-sync.","solutions":["Check dmesg/system logs and free space — ENOSPC/EIO at fsync usually means real storage trouble","Retry once after clearing space if the kind is StorageFull; otherwise fail the snapshot operation — data is not durable","Use a healthy local filesystem for snapshot targets; avoid FUSE/network mounts for durability-critical files","Alert on this error: it invalidates the crash-recovery guarantee of the snapshot just written"],"exampleFix":"// before\nSyncTarget::File(&file).sync_all()?; // EIO: dying disk\n\n// after\nif let Err(e) = SyncTarget::File(&file).sync_all() {\n    if e.kind() == std::io::ErrorKind::StorageFull {\n        // free space then retry once\n        cleanup_old_snapshots()?;\n        SyncTarget::File(&file).sync_all()?;\n    } else {\n        return Err(e); // durability violated — surface loudly\n    }\n}","handlingStrategy":"retry","validationCode":"// Pre-flight headroom so writeback-time ENOSPC is unlikely:\nfn disk_headroom(p: &Path) -> bool {\n    // platform-specific statvfs; simplify: require > 2x expected snapshot size\n    true\n}","typeGuard":null,"tryCatchPattern":"let mut attempt = 0;\nloop {\n    match SyncTarget::File(&file).sync_all() {\n        Ok(()) => break,\n        Err(ref e) if e.kind() == std::io::ErrorKind::StorageFull && attempt == 0 => {\n            cleanup_old_snapshots()?; attempt += 1; // free space, retry once\n        }\n        Err(e) => return Err(e), // durability violated: alert, do not mask\n    }\n}","preventionTips":["Keep free disk space well above worst-case snapshot size","Alert on any file-fsync failure — the snapshot is not durable","Use healthy local disks; avoid FUSE/network mounts for durability-critical files","Retry only transient kinds (StorageFull after cleanup); never mask EIO"],"tags":["filesystem","fsync","durability","rust","snapshot","spacetimedb"],"backgroundTag":"fsync-failed","analyzedSha":"9e0d92412ff2248f401a8ad12d535f2b5ac30912","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}