{"record":{"id":"c4ca8bc77b0f35af","repo":"clockworklabs/SpacetimeDB","slug":"fsync-failed","errorCode":null,"errorMessage":"fsync failed","messagePattern":"fsync failed","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/commitlog/src/stream/common.rs","lineNumber":51,"sourceCode":"    fn open_segment_reader_async(\n        &self,\n        offset: u64,\n    ) -> impl Future<Output = io::Result<Self::AsyncSegmentReader>> + Send;\n}\n\npub trait AsyncFsync {\n    fn fsync(&self) -> impl Future<Output = ()> + Send;\n}\n\nimpl<T: AsyncWrite + AsyncFsync + Send + Sync> AsyncFsync for tokio::io::BufWriter<T> {\n    async fn fsync(&self) {\n        self.get_ref().fsync().await\n    }\n}\n\nimpl AsyncFsync for tokio::fs::File {\n    async fn fsync(&self) {\n        self.sync_data().await.expect(\"fsync failed\")\n    }\n}\n\npub trait AsyncLen: AsyncSeek + Unpin + Send {\n    fn segment_len(&mut self) -> impl Future<Output = io::Result<u64>> + Send\n    where\n        Self: Sized,\n    {\n        async { spacetimedb_fs_utils::compression::segment_len(self).await }\n    }\n}\n\nimpl<T: AsyncWrite + AsyncLen + Send> AsyncLen for tokio::io::BufWriter<T> {\n    async fn segment_len(&mut self) -> io::Result<u64> {\n        self.get_mut().segment_len().await\n    }\n}\n","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/9e0d92412ff2248f401a8ad12d535f2b5ac30912/crates/commitlog/src/stream/common.rs#L33-L69","documentation":"Panic inside SpacetimeDB's async commitlog stream layer: the `AsyncFsync` implementation for `tokio::fs::File` calls `sync_data().await.expect(\"fsync failed\")`. Any I/O error from the OS `fsync`/`fdatasync` call — hardware error, dropped network storage, file already closed — becomes a panic instead of a `Result`, taking down the task that awaited it.","triggerScenarios":"Any code path that syncs a commitlog segment file: segment rotation, durability flushes, or explicit sync of the log writer. `sync_data()` returns Err on EIO/EBADF/ENOSPC (e.g. failing disk, file closed early, thin-provisioned volume out of space), and the `expect` converts it to a panic inside the async task.","commonSituations":"Failing or ejecting disk; container overlay filesystem errors; NFS/SMB disconnect mid-sync; disk quota hit during fdatasync; test harnesses closing files early. Because the panic occurs in a `Send + Sync` impl shared by writer tasks, it usually cascades into the 'log worker panicked' / commitlog poisoning panics (errors 603/605).","solutions":["Check disk health immediately (`dmesg | grep -i 'i/o error'`, SMART status) — an fsync failure usually means real data risk; restore the database from snapshot after fixing hardware.","Ensure the data directory is on reliable local storage, not a flaky network mount, and that quotas/limits leave headroom for sync.","Restart the node after the storage fault clears; treat the last partially synced segment as suspect (replay from the previous good snapshot).","If you control the code, prefer returning `io::Result` from custom `AsyncFsync` impls instead of panicking, so上层 durability code can handle failures gracefully."],"exampleFix":"// before: panic on fsync error inside the trait impl\nimpl AsyncFsync for tokio::fs::File {\n    async fn fsync(&self) { self.sync_data().await.expect(\"fsync failed\") }\n}\n\n// after (for custom storage): surface the error\nimpl AsyncFsync for MyFile {\n    async fn fsync(&self) {\n        if let Err(e) = self.sync_data().await { /* log and propagate */ }\n    }\n}","handlingStrategy":"fallback","validationCode":"// Pre-check before relying on sync: ensure the fd is valid and space remains\nif available_space(data_dir)? < MIN_FREE_BYTES { return Err(io::Error::new(ErrorKind::StorageFull, \"low disk\")); }","typeGuard":null,"tryCatchPattern":"let res = future::FutureExt::catch_unwind(std::pin::pin!(file.fsync()));\nif res.await.is_err() { /* storage fault: stop writer, escalate, restart process */ }","preventionTips":["Alert on disk errors in dmesg/SMART before fsync fails for real.","Keep DB files off network mounts prone to transient drops.","Treat fsync panics as data-integrity events: restore from snapshot after remediation, don't just restart."],"tags":["rust","spacetimedb","fsync","disk-io","tokio","panic","storage"],"backgroundTag":"fsync-failed","analyzedSha":"9e0d92412ff2248f401a8ad12d535f2b5ac30912","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}