{"record":{"id":"e3136bf165174bfc","repo":"apache/cassandra","slug":"invalid-remote-counter-shard-detected","errorCode":null,"errorMessage":"invalid remote counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in count; will pick highest to self-heal on compaction","messagePattern":"invalid remote counter shard detected; \\((.+?), (.+?), (.+?)\\) and \\((.+?), (.+?), (.+?)\\) differ only in count; will pick highest to self-heal on compaction","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/db/context/CounterContext.java","lineNumber":513,"sourceCode":"\n        if (leftState.isLocal() || rightState.isLocal())\n        {\n            // Local id and at least one is a local shard.\n            if (leftState.isLocal() && rightState.isLocal())\n                return Relationship.DISJOINT;\n            else // only one is local - keep that one\n                return leftState.isLocal() ? Relationship.GREATER_THAN : Relationship.LESS_THAN;\n        }\n\n        // both are remote shards\n        if (leftClock == rightClock)\n        {\n            // We should never see non-local shards w/ same id+clock but different counts. However, if we do\n            // we should \"heal\" the problem by being deterministic in our selection of shard - and\n            // log the occurrence so that the operator will know something is wrong.\n            if (leftCount != rightCount && CompactionManager.isCompactor(Thread.currentThread()))\n            {\n                logger.warn(\"invalid remote counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in \"\n                            + \"count; will pick highest to self-heal on compaction\",\n                            leftState.getCounterId(), leftClock, leftCount,\n                            rightState.getCounterId(), rightClock, rightCount);\n            }\n\n            if (leftCount > rightCount)\n                return Relationship.GREATER_THAN;\n            else if (leftCount == rightCount)\n                return Relationship.EQUAL;\n            else\n                return Relationship.LESS_THAN;\n        }\n        else\n        {\n            if ((leftClock >= 0 && rightClock > 0 && leftClock >= rightClock)\n                    || (leftClock < 0 && (rightClock > 0 || leftClock < rightClock)))\n                return Relationship.GREATER_THAN;\n            else","sourceCodeStart":495,"sourceCodeEnd":531,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/db/context/CounterContext.java#L495-L531","documentation":"Same self-healing logic as the global shard case, but for remote counter shards: two shards with the same counterId and clock but different counts. Per the code comment this should never happen; when it does on a compactor thread, the highest count is deterministically chosen and the occurrence is logged so operators know something went wrong.","triggerScenarios":"Compaction (isCompactor thread) comparing remote (non-local) counter shards with equal id+clock but unequal counts — caused by lost/corrupt sstables, improper manual sstable manipulation, or restored data that is inconsistent within a single sstable set.","commonSituations":"Restoring partial backups of counter tables; sstable corruption; copying sstables between nodes/datacenters manually; best_effort disk failure policy masking data loss.","solutions":["Run nodetool verify/scrub on the node to detect corrupt sstables and run repair on the counter table.","Investigate backup/restore procedures — do not mix sstables from different restore points.","Replace best_effort disk policy with stop or die to catch storage failures deterministically.","If counts are business-critical, compare against replicas and consider re-writing the counter from application-level data."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// audit restore hygiene for counter tables\nif (restoredSstablesMixedSnapshots) {\n    throw new IllegalStateException(\"Refusing mixed-snapshot sstable restore for counter table\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Do not copy sstables manually between nodes or DCs.","Restore counter tables only from single, complete snapshots.","Use stop/die disk failure policy to surface storage failures.","Run nodetool verify after any suspicious storage event.","Alert on this warning — it signals impossible counter state was healed."],"tags":["cassandra","counters","data-corruption","compaction","sstable"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}