apache/cassandra · warning · SegmentReadException

Encountered bad header at position %d of commit log %s, with

Error message

Encountered bad header at position %d of commit log %s, with invalid CRC. The end of segment marker should be zero.

What it means

During commit log replay, CommitLogSegmentReader.readSyncMarker validates each sync marker. When the marker CRC matches but the recorded 'end' is the segment end marker (0) while trailing bytes/CRC are nonzero, the header is deemed invalid and SegmentReadException(tolerable=true) is thrown: 'The end of segment marker should be zero.' It indicates a truncated or corrupted segment tail.

Source

Thrown at src/java/org/apache/cassandra/db/commitlog/CommitLogSegmentReader.java:203

            // The next marker position and CRC value are not written atomically, so it is possible for the latter to 
            // still be zero after the former has been finalized, even though the mutations that follow it are valid.
            // When there is no compression or encryption enabled, we can ignore a sync marker CRC mismatch and defer 
            // to the per-mutation CRCs, which may be preferable to preventing startup altogether.
            if (allowSkipSyncMarkerCrc
                && descriptor.compression == null && !descriptor.getEncryptionContext().isEnabled()
                && filecrc == 0 && end != 0)
            {
                logger.warn("Skipping sync marker CRC check at position {} (end={}, calculated crc={}) of commit log {}." +
                            "Using per-mutation CRC checks to ensure correctness...",
                            offset, end, crc.getValue(), reader.getPath());
                return end;
            }

            if (end != 0 || filecrc != 0)
            {
                String msg = String.format("Encountered bad header at position %d of commit log %s, with invalid CRC. " +
                             "The end of segment marker should be zero.", offset, reader.getPath());
                throw new SegmentReadException(msg, true);
            }
            return -1;
        }
        else if (end < offset || end > reader.length())
        {
            String msg = String.format("Encountered bad header at position %d of commit log %s, with bad position but valid CRC", offset, reader.getPath());
            throw new SegmentReadException(msg, false);
        }
        return end;
    }

    public static class SegmentReadException extends IOException
    {
        public final boolean invalidCrc;

        public SegmentReadException(String msg, boolean invalidCrc)
        {
            super(msg);

View on GitHub (pinned to 88fd0f6a0e)

Solutions

  1. Let Cassandra skip the corrupt tail: replay handles SegmentReadException with tolerable=true by stopping at the last valid marker.
  2. Remove the affected segment from commitlog/ if replay is blocked and data loss is acceptable (it will be restored from SSTables).
  3. Restore from backup/SSTables and run a repair (nodetool repair) to re-sync replicas.
  4. Check disk health (dmesg, SMART) if corruption recurs.
Defensive patterns

Strategy: try-catch

Try / catch

try {
    end = CommitLogSegmentReader.readSyncMarker(reader, offset);
} catch (SegmentReadException e) {
    if (e.tolerable)
        logger.warn("Corrupt segment tail, stopping replay at last valid marker: {}", e.getMessage());
    else
        throw e;
}

Prevention

When it happens

Trigger: Replaying a commit log whose final sync marker region contains a nonzero end marker or nonzero CRC where zeros are required — typically due to a crash mid-write, torn page, or manual file truncation at a non-marker boundary.

Common situations: Node crash without clean shutdown; disk full during segment close; file copied/synced mid-write (e.g. backup of live segment); corrupted storage.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/b320c43b44eb2595. Report an issue: GitHub.