{"record":{"id":"1155d7f91c77d3cc","repo":"clockworklabs/SpacetimeDB","slug":"failed-to-flush-segment-upon-rotation","errorCode":null,"errorMessage":"failed to flush segment upon rotation","messagePattern":"failed to flush segment upon rotation","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/commitlog/src/commitlog.rs","lineNumber":317,"sourceCode":"    ///\n    /// - `transactions` exceeds [u16::MAX] elements\n    ///\n    /// - [Self::flush] or writing to the underlying [Writer] fails\n    ///\n    ///   This is likely caused by some storage issue. As we cannot tell with\n    ///   certainty how much data (if any) has been written, the internal state\n    ///   becomes invalid and thus a panic is raised.\n    ///\n    /// - [Self::sync] panics (called when rotating segments)\n    pub fn commit<U: Into<Transaction<T>>>(\n        &mut self,\n        transactions: impl IntoIterator<Item = U>,\n    ) -> io::Result<Option<Committed>> {\n        self.panicked = true;\n        let writer = &mut self.head;\n        let committed = writer.commit(transactions)?;\n        if writer.len() >= self.opts.max_segment_size {\n            self.flush().expect(\"failed to flush segment upon rotation\");\n            self.sync();\n            self.start_new_segment()?;\n        }\n        self.panicked = false;\n\n        Ok(committed)\n    }\n\n    pub fn transactions_from<'a, D>(\n        &self,\n        offset: u64,\n        decoder: &'a D,\n    ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a + use<'a, D, R, T>\n    where\n        D: Decoder<Record = T>,\n        D::Error: From<error::Traversal>,\n        R: 'a,\n        T: 'a,","sourceCodeStart":299,"sourceCodeEnd":335,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/524b4487d949b61a07d4f39c862d1290259dfd20/crates/commitlog/src/commitlog.rs#L299-L335","documentation":"SpacetimeDB's commit log stores transactions in size-bounded segment files. When `Commitlog::commit` grows the head segment past `opts.max_segment_size`, it must flush (write buffered data and fsync) before starting a new segment; this `expect` fires when that flush returns an I/O error. Because a failed flush leaves unknown how much data reached disk, the log poisons its state (note the `panicked` flag) and aborts rather than continue with invalid state.","triggerScenarios":"Calling `Commitlog::commit(transactions)` repeatedly until `writer.len() >= opts.max_segment_size` triggers rotation; the panic happens only if the subsequent `self.flush()` errors — ENOSPC (disk full), EIO (failing device), EACCES/EBADF on the segment file, or an fsync error surfaced by the OS.","commonSituations":"Host disk filling up because segment retention keeps too much data; running the node on a volume with an exhausted quota; failing disk or filesystem corruption; a container with a small tmpfs or read-only mount for the data directory.","solutions":["Check free space and quotas on the commitlog data directory (df -h, container disk limits); ENOSPC is the most common cause — free space or expand the volume.","Inspect dmesg/journalctl for I/O errors on the underlying device and run smartctl/fsck if the disk is unhealthy.","Verify the data directory and segment files are writable by the spacetimedb process user (permissions, read-only mounts).","Restart the node after remediation; do not reuse in-memory commitlog state after this panic — it is intentionally poisoned.","Review max_segment_size and retention settings so the volume has headroom."],"exampleFix":"// before: rotation flush failure aborts the process inside commit()\nlog.commit(txs)?;\n\n// after: catch the panic at the call site and surface it as an error\nlet committed = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| log.commit(txs)))\n    .map_err(|_| io::Error::new(io::ErrorKind::Other, \"commitlog flush failed during rotation\"))??;","handlingStrategy":"validation","validationCode":"// Before committing, check the segment directory has room for a full rotation\n// (nix crate): let vfs = nix::sys::statvfs::statvfs(data_dir)?;\n// let free = vfs.blocks_available() as u64 * vfs.block_size() as u64;\n// assert!(free > max_segment_size + headroom);","typeGuard":null,"tryCatchPattern":"let r = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| log.commit(txs))); match r { Ok(res) => { /* process Ok(res) */ } Err(_) => { /* log dir + disk state; restart node — state is poisoned */ } }","preventionTips":["Monitor free space on the commitlog volume and alert well before it fills.","Keep data directories on healthy, writable local volumes; avoid read-only mounts and tiny tmpfs sizes in containers.","Size max_segment_size and retention against available disk capacity.","Treat any commitlog panic as fatal: restart the process instead of reusing in-memory state."],"tags":["rust","commitlog","fsync","disk-io","segment-rotation","spacetimedb"],"backgroundTag":"disk-fsync-failure","analyzedSha":"524b4487d949b61a07d4f39c862d1290259dfd20","analyzedAt":"2026-08-16T23:58:54.611Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}