{"record":{"id":"68cc224558efe5e4","repo":"clockworklabs/SpacetimeDB","slug":"repo-too-many-empty-segments","errorCode":null,"errorMessage":"repo {}: too many empty segments: {}","messagePattern":"repo (.+?): too many empty segments: (.+?)","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"warning","filePath":"crates/commitlog/src/commitlog.rs","lineNumber":476,"sourceCode":"            mut segment_reader,\n        }) = open_newest_non_empty_segment(&repo)?\n        else {\n            return Ok(None);\n        };\n        let offset_index = repo.get_offset_index(segment_offset).ok();\n        match segment::Metadata::extract(segment_offset, &mut segment_reader, offset_index.as_ref()) {\n            // Segment is intact.\n            Ok(metadata) if empty_segments <= 1 => {\n                assert!(\n                    !metadata.tx_range.is_empty(),\n                    \"segment was promised to be non-empty but contains zero transactions\"\n                );\n                Ok(Some(CommittedMeta::Complete { metadata }))\n            }\n            // Segment is good, but there are too many empty segments.\n            Ok(metadata) => Ok(Some(CommittedMeta::Prefix {\n                metadata,\n                error: io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    format!(\"repo {}: too many empty segments: {}\", repo, empty_segments),\n                ),\n            })),\n            // Segment is non-empty, but first commit is corrupt.\n            Err(error::SegmentMetadata::InvalidCommit { sofar, source }) if sofar.tx_range.is_empty() => {\n                Err(io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    format!(\n                        \"repo {}: first commit in the most recent segment is corrupt: {}\",\n                        repo, source\n                    ),\n                ))\n            }\n            // Some prefix of the segment is good.\n            Err(error::SegmentMetadata::InvalidCommit { sofar, source }) => Ok(Some(CommittedMeta::Prefix {\n                metadata: sofar,\n                error: source,","sourceCodeStart":458,"sourceCodeEnd":494,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/9e0d92412ff2248f401a8ad12d535f2b5ac30912/crates/commitlog/src/commitlog.rs#L458-L494","documentation":"Recovery-time diagnostic from CommittedMeta::extract. While locating the most recent non-empty segment, the opener walks backwards over trailing empty segments (created but never given data, usually after a crash); if it skips more than one, the recovered metadata is still returned but wrapped as CommittedMeta::Prefix with this InvalidData error attached instead of CommittedMeta::Complete. The log stays usable up to the returned metadata - this signals accumulated crash debris, not corruption of the returned data.","triggerScenarios":"A node that crashed and restarted several times in a row, each restart creating a fresh empty segment before dying again; opening the commitlog then counts empty_segments >= 2 and reports the Prefix with this message.","commonSituations":"Crash loops (OOM killer, failing disk, bad deploy) that repeatedly roll over to a new segment without committing anything; leftover empty segment files from earlier runs in the data directory.","solutions":["Usually self-healing: once the node commits successfully the empty tail stops growing and the condition disappears on next open.","If it recurs, find out why the process dies before its first commit (node logs, OOM, disk full).","In a dev environment, wiping and re-initializing the commitlog directory clears the debris."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"match committed_meta(repo)? {\n    Some(CommittedMeta::Complete { metadata }) => use_metadata(metadata),\n    Some(CommittedMeta::Prefix { metadata, error }) => {\n        tracing::warn!(\"log has trailing damage/empties: {error}\");\n        use_metadata(metadata) // proceed with the good prefix\n    }\n    None => { /* empty log */ }\n}","preventionTips":["Fix crash loops quickly - each crash before the first commit adds another empty segment.","Let nodes shut down gracefully so segments are never left empty at the tail.","Treat repeated occurrences of this warning as 'the node never succeeds in committing' and investigate availability."],"tags":["rust","commitlog","recovery","segments","crash-loop"],"backgroundTag":"crash-recovery-artifacts","analyzedSha":"9e0d92412ff2248f401a8ad12d535f2b5ac30912","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}