{"record":{"id":"ea4742ce5a98fc20","repo":"apache/cassandra","slug":"unrecoverable-error-while-scrubbing-s-scrubbing-c","errorCode":null,"errorMessage":"Unrecoverable error while scrubbing %s.Scrubbing cannot continue. The sstable will be marked for deletion. You can attempt manual recovery from the pre-scrub snapshot. You can also run nodetool repair to transfer the data from a healthy replica, if any.","messagePattern":"Unrecoverable error while scrubbing (.+?)\\.Scrubbing cannot continue\\. The sstable will be marked for deletion\\. You can attempt manual recovery from the pre-scrub snapshot\\. You can also run nodetool repair to transfer the data from a healthy replica, if any\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/io/sstable/format/bti/BtiTableScrubber.java","lineNumber":237,"sourceCode":"                        badPartitions++;\n                        if (!seekToNextPartition())\n                            break;\n                    }\n                }\n                else\n                {\n                    throwIfCannotContinue(key, th);\n\n                    badPartitions++;\n                    if (indexIterator != null)\n                    {\n                        outputHandler.warn(\"Partition starting at position %d is unreadable; skipping to next\", dataStart);\n                        if (!seekToNextPartition())\n                            break;\n                    }\n                    else\n                    {\n                        outputHandler.warn(\"Unrecoverable error while scrubbing %s.\" +\n                                           \"Scrubbing cannot continue. The sstable will be marked for deletion. \" +\n                                           \"You can attempt manual recovery from the pre-scrub snapshot. \" +\n                                           \"You can also run nodetool repair to transfer the data from a healthy replica, if any.\",\n                                           sstable);\n                        // There's no way to resync and continue. Give up.\n                        break;\n                    }\n                }\n            }\n        }\n    }\n\n\n    private boolean indexAvailable()\n    {\n        return indexIterator != null && !indexIterator.isExhausted();\n    }\n","sourceCodeStart":219,"sourceCodeEnd":255,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/io/sstable/format/bti/BtiTableScrubber.java#L219-L255","documentation":"During a BTI sstable scrub, if an unrecoverable error occurs while reading a partition (scrub cannot resync to the next partition), the scrubber logs this warning and stops scrubbing the sstable. The sstable is then marked for deletion and the operator is pointed at the pre-scrub snapshot or nodetool repair as recovery paths. This is a warning-level output, not an exception: scrub gives up on this sstable rather than crashing the node.","triggerScenarios":"scrubInternal encounters a partition whose data cannot be read or skipped in a way that lets the scrubber seek to the next partition — e.g. corrupted partition index/row data in a BTI sstable such that seekToNextPartition() also fails.","commonSituations":"Disk corruption or bit rot on an sstable; failed writes leaving a truncated/corrupt BTI data file; hardware issues; running scrub on sstables damaged by an earlier crash or bad disk sector.","solutions":["Restore the sstable from the pre-scrub snapshot (nodetool listsnapshots; copy from the snapshot dir) and re-attempt or replace the data","Run nodetool repair to stream the missing data from a healthy replica","If replicas hold the data, let the corrupted sstable be marked for deletion and rely on repair/anti-entropy to restore consistency","Check disk health (dmesg, SMART) and replace failing hardware before re-running scrub"],"exampleFix":"// before (damaged sstable, no snapshot)\n$ nodetool scrub keyspace1 standard1  // scrub aborts, sstable deleted\n// after\n$ nodetool listsnapshots  # find pre-scrub snapshot\n$ nodetool repair keyspace1  // re-stream data from healthy replicas","handlingStrategy":"fallback","validationCode":"// before scrub: ensure snapshot exists and replicas are healthy\nboolean hasSnapshot = java.nio.file.Files.exists(java.nio.file.Path.of(sstableDir, \"snapshots\", \"pre-scrub\"));\nboolean replicasHealthy = Keyspace.open(keyspace).getReplicationStrategy()\n        .getNaturalReplicasForToken(sstable.first.getToken()).stream()\n        .allMatch(ep -> FailureDetector.instance.isAlive(ep));\nif (!hasSnapshot || !replicasHealthy) throw new IllegalStateException(\"Take a snapshot and verify replicas before scrubbing\");","typeGuard":null,"tryCatchPattern":"// scrub is invoked via nodetool/StorageService; wrap at orchestration level\ntry {\n    Scrubber scrubber = sstable.getScrubber(...);\n    scrubber.scrub();\n} catch (IOError e) {\n    logger.warn(\"Scrub gave up on {} (unrecoverable). Restore from pre-scrub snapshot or run repair.\", sstable, e);\n}","preventionTips":["Always take a snapshot (nodetool snapshot) before running scrub","Keep RF >= 3 so nodetool repair can restore data lost from a corrupt sstable","Monitor disk health (SMART) and filesystem errors; scrub is a symptom of corruption, not a fix for hardware","Run scrub during low traffic and inspect outputHandler warnings for unreadable partitions"],"tags":["scrub","sstable","data-corruption","storage"],"backgroundTag":"checksum-mismatch","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}