apache/cassandra · error

An error occurred while scrubbing the partition with key '%s

Error message

An error occurred while scrubbing the partition with key '%s'.  Skipping corrupt data in counter tables will result in undercounts for the affected counters (see CASSANDRA-2759 for more details), so by default the scrub will stop at this point.  If you would like to skip the row anyway and continue scrubbing, re-run the scrub with the --skip-corrupted option.

What it means

nodetool scrub encountered an unreadable/corrupt partition on a counter table. Because skipping corrupt data in counter tables permanently undercounts affected counters (CASSANDRA-2759), the scrubber warns and aborts (throws IOError) unless --skip-corrupted was passed. This is a deliberate safety stop to prevent silent counter data loss.

Source

Thrown at src/java/org/apache/cassandra/io/sstable/format/SortedTableScrubber.java:352

    private void saveOutOfOrderPartition(DecoratedKey prevKey, DecoratedKey key, UnfilteredRowIterator iterator)
    {
        // TODO bitch if the row is too large?  if it is there's not much we can do ...
        outputHandler.warn("Out of order partition detected (%s found after %s)", keyString(key), keyString(prevKey));
        outOfOrder.add(ImmutableBTreePartition.create(iterator));
    }

    protected static void throwIfFatal(Throwable th)
    {
        if (th instanceof Error && !(th instanceof AssertionError || th instanceof IOError))
            throw (Error) th;
    }

    protected void throwIfCannotContinue(DecoratedKey key, Throwable th)
    {
        if (isCommutative && !options.skipCorrupted)
        {
            outputHandler.warn("An error occurred while scrubbing the partition with key '%s'.  Skipping corrupt " +
                               "data in counter tables will result in undercounts for the affected " +
                               "counters (see CASSANDRA-2759 for more details), so by default the scrub will " +
                               "stop at this point.  If you would like to skip the row anyway and continue " +
                               "scrubbing, re-run the scrub with the --skip-corrupted option.",
                               keyString(key));
            throw new IOError(th);
        }
    }


    public static class ScrubInfo extends CompactionInfo.Holder
    {
        private final RandomAccessReader dataFile;
        private final SSTableReader sstable;
        private final TimeUUID scrubCompactionId;
        private final Lock fileReadLock;

        public ScrubInfo(RandomAccessReader dataFile, SSTableReader sstable, Lock fileReadLock)

View on GitHub (pinned to 88fd0f6a0e)

Solutions

1. First back up the affected SSTables, then re-run scrub with --skip-corrupted if you accept undercounts for the affected counters.
2. Attempt to restore affected counters: re-run counter increments or rebuild the table from application data/other replicas.
3. Check replicas: if other replicas have healthy data, run repair (nodetool repair) to restore the partition instead of skipping.
4. Investigate root cause of corruption (disk health, filesystem, CASSANDRA-version bugs) to prevent recurrence.

Example fix

// before: scrub aborts on counter table corruption
nodetool scrub <keyspace> <counter_table>
// after: accept undercounts and continue (backup SSTables first!)
nodetool scrub --skip-corrupted <keyspace> <counter_table>
nodetool repair <keyspace> <counter_table>

# Detect corruption before scrub so you can plan (backup first!)
nodetool verify <keyspace> <counter_table>   # reports corrupt SSTables non-destructively
# Snapshot before any scrub:
nodetool snapshot <keyspace> --table <counter_table>

// CLI-level: scrub aborts with IOError; handle by planning recovery
// nodetool scrub ks counter_tbl  -> aborts on counter corruption
// Fallback sequence: snapshot -> scrub --skip-corrupted -> repair -> validate counters from app data

When it happens

Trigger: Running nodetool scrub on a counter table containing a corrupted partition (torn write, disk corruption, bad SSTable) without --skip-corrupted.

Common situations: Recovering from disk failures or crashed hardware; scrubbing after unclean shutdowns on counter-heavy tables; repair of corrupt SSTables found by nodetool verify.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• An error occurred while scrubbing the partition with key '%s
• Error reading partition %s (stacktrace follows):
• Retry failed too. Skipping to next partition (retry's stackt
• Not enough bytes to header of the comparator part of compone
• Not enough bytes to read comparator name of component %s

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/2abeca5042d3ff7c. Report an issue: GitHub.