{"record":{"id":"1155d7f91c77d3cc","repo":"clockworklabs/SpacetimeDB","slug":"failed-to-flush-segment-upon-rotation","errorCode":null,"errorMessage":"failed to flush segment upon rotation","messagePattern":"failed to flush segment upon rotation","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"crates/commitlog/src/commitlog.rs","lineNumber":317,"sourceCode":"    ///\n    /// - `transactions` exceeds [u16::MAX] elements\n    ///\n    /// - [Self::flush] or writing to the underlying [Writer] fails\n    ///\n    ///   This is likely caused by some storage issue. As we cannot tell with\n    ///   certainty how much data (if any) has been written, the internal state\n    ///   becomes invalid and thus a panic is raised.\n    ///\n    /// - [Self::sync] panics (called when rotating segments)\n    pub fn commit<U: Into<Transaction<T>>>(\n        &mut self,\n        transactions: impl IntoIterator<Item = U>,\n    ) -> io::Result<Option<Committed>> {\n        self.panicked = true;\n        let writer = &mut self.head;\n        let committed = writer.commit(transactions)?;\n        if writer.len() >= self.opts.max_segment_size {\n            self.flush().expect(\"failed to flush segment upon rotation\");\n            self.sync();\n            self.start_new_segment()?;\n        }\n        self.panicked = false;\n\n        Ok(committed)\n    }\n\n    pub fn transactions_from<'a, D>(\n        &self,\n        offset: u64,\n        decoder: &'a D,\n    ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a + use<'a, D, R, T>\n    where\n        D: Decoder<Record = T>,\n        D::Error: From<error::Traversal>,\n        R: 'a,\n        T: 'a,","sourceCodeStart":299,"sourceCodeEnd":335,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/9e0d92412ff2248f401a8ad12d535f2b5ac30912/crates/commitlog/src/commitlog.rs#L299-L335","documentation":"Panic in SpacetimeDB's commit log when a segment is rotated. After `commit()` writes transactions, if the current segment exceeds `opts.max_segment_size`, the log flushes, syncs, and starts a new segment; a flush I/O error here hits `.expect(\"failed to flush segment upon rotation\")`. The code intentionally panics because after a partial write the log's internal state is invalid — it cannot know how much data reached disk, and continuing would corrupt the log.","triggerScenarios":"Calling `CommitLog::commit` repeatedly until the head segment reaches `max_segment_size`, at which point `flush()` returns an I/O error: ENOSPC (disk full), EIO, permission loss on the data directory, or the underlying file being closed/removed underneath the process.","commonSituations":"Long-running SpacetimeDB host/minion whose database directory fills the disk; container volume limits reached; storage path on a network mount that dropped; running out of inodes; another process deleting segment files. After this panic the commitlog is poisoned (`self.panicked = true`) and every subsequent commit also panics by design.","solutions":["Free disk space on the volume holding the database/commitlog directory (check `df -h` and inode usage with `df -i`), then restart the SpacetimeDB node — do not try to resume the poisoned in-process log.","Verify permissions and open file descriptors on the data directory; ensure no external process prunes segment files.","Configure monitoring/disk quotas so the volume cannot silently fill; size `max_segment_size` and retention so rotation pressure is predictable.","If the flush failure was transient (e.g. NFS blip), a full restart replays from the last synced segment — inspect logs for torn writes and restore from snapshot if replay fails."],"exampleFix":"// before: letting the disk fill until the commitlog panics on rotation\nhost.commit(tx).unwrap();\n\n// after: guard free space before committing (ops-level check)\nif free_bytes(db_dir) < MIN_FREE_BYTES { return Err(HostError::DiskLow); }\nhost.commit(tx)?;","handlingStrategy":"fallback","validationCode":"// Pre-flight: refuse to commit when the log volume is nearly full\nfn can_commit(db_dir: &Path) -> bool {\n    fs2::available_space(db_dir).map(|b| b > MIN_FREE_BYTES).unwrap_or(false)\n}","typeGuard":null,"tryCatchPattern":"// Rust: contain the deliberate poison-panic so the caller can degrade\nlet r = std::panic::catch_unwind(AssertUnwindSafe(|| log.commit(txns)));\nmatch r { Ok(res) => res?, Err(p) => { /* mark log dead, restart process / failover */ } }","preventionTips":["Monitor free space and inodes on the commitlog volume; alert well before it fills.","Keep the database directory on reliable local storage; never let external jobs prune segment files.","Treat any commitlog panic as 'process must restart' — the in-memory log is poisoned by design.","Run periodic snapshots so recovery does not depend on the tail segments."],"tags":["rust","spacetimedb","commitlog","disk-full","io","panic","storage"],"backgroundTag":"disk-write-failed","analyzedSha":"9e0d92412ff2248f401a8ad12d535f2b5ac30912","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}