{"record":{"id":"efa7176311309045","repo":"clockworklabs/SpacetimeDB","slug":"all-zeroes-commit-header","errorCode":null,"errorMessage":"all-zeroes commit header","messagePattern":"all-zeroes commit header","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"crates/commitlog/src/stream/writer.rs","lineNumber":306,"sourceCode":"                // The stream is exhausted, break the outer loop.\n                trace!(\"eof\");\n                return Ok(AppendInnerResult::StreamExhausted);\n            };\n            if buf.starts_with(&segment::MAGIC) {\n                // New segment, break inner loop.\n                trace!(\"segment marker\");\n                return Ok(AppendInnerResult::SegmentMarker);\n            }\n\n            // Read the header, so we can determine the size of the commit.\n            if read_exact(stream, &mut self.commit_buf.header).await?.is_eof() {\n                return Ok(AppendInnerResult::StreamExhausted);\n            }\n            let Some(commit_header) = commit::Header::decode(&self.commit_buf.header[..])\n                .inspect_err(|e| warn!(\"failed to decode commit header: {e}\"))?\n            else {\n                // Nb. eof handled above.\n                return Err(io::Error::new(io::ErrorKind::InvalidData, \"all-zeroes commit header\"));\n            };\n\n            // Read the rest of the commit.\n            self.commit_buf.body.resize(\n                commit_header.len as usize + CHECKSUM_LEN[current_segment.header.checksum_algorithm as usize],\n                0,\n            );\n            stream.read_exact(&mut self.commit_buf.body).await?;\n            // Decode the commit and verify its checksum.\n            let commit = StoredCommit::decode(&mut self.commit_buf.as_reader(bytes_written))\n                .inspect_err(|e| warn!(\"failed to decode commit: {e}\"))?\n                .expect(\"commit decode cannot return `None` because we already decoded the header\");\n\n            // Check that the commit offset is what we expect.\n            let expected_offset = self\n                .last_written_tx_range\n                .as_ref()\n                .map(|range| range.end)","sourceCodeStart":288,"sourceCodeEnd":324,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/9e0d92412ff2248f401a8ad12d535f2b5ac30912/crates/commitlog/src/stream/writer.rs#L288-L324","documentation":"While decoding the mirrored stream, the fixed-size commit header was read successfully but decoded as all zeroes, which commit::Header::decode rejects (EOF was already handled just above). Zero regions typically come from preallocated-but-unwritten file space or a torn write on the producer side, so the stream contains filler instead of commit data.","triggerScenarios":"The producer streamed a live segment up to its fallocated length instead of its committed length; the source crashed mid-write and the tail zeroes were included; zero padding was deliberately inserted into the stream.","commonSituations":"Mirroring by reading a segment file to file-size rather than to the last valid commit; snapshotting a file while it is being appended to; tail of a preallocated segment included by an off-by-length error.","solutions":["Stream only up to the last valid commit: bound the read to the committed length from segment metadata, not the file size","Trim trailing invalid data on the source before streaming (the OnTrailingData hook used at StreamWriter::create exists for this)","On the consumer, stop the stream at the first all-zero header instead of erroring, when partial data is acceptable","Fix the producer to publish the committed offset alongside the data"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// On the producer side: stream only up to the committed length, not the file size\nlet committed_len = segment_metadata.size_in_bytes + segment::Header::LEN as u64;\nlet mut limited = (&mut file).take(committed_len); // never read preallocated zero tail\nstream_from(&mut limited).await;","typeGuard":"fn is_all_zero_header(err: &io::Error) -> bool {\n    err.kind() == io::ErrorKind::InvalidData && err.to_string().contains(\"all-zeroes commit header\")\n}","tryCatchPattern":"match writer.append_all(&mut stream, &mut progress).await {\n    Ok(done) => Ok(done),\n    Err(e) if is_all_zero_header(&e) => {\n        // tail of preallocated/torn data reached: treat stream as ended, keep what was written\n        Ok(StreamDone::Exhausted)\n    }\n    Err(e) => Err(e),\n}","preventionTips":["Publish the committed offset with the stream; bound reads to it","Trim trailing invalid data on the source before mirroring (OnTrailingData)","Never stream a live segment to its fallocated length"],"tags":["rust","commitlog","streaming","torn-write","invalid-data"],"backgroundTag":"partial-write-corruption","analyzedSha":"9e0d92412ff2248f401a8ad12d535f2b5ac30912","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}