{"record":{"id":"137c705b35ae8d80","repo":"apache/cassandra","slug":"invalid-global-counter-shard-detected","errorCode":null,"errorMessage":"invalid global counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in count; will pick highest to self-heal on compaction","messagePattern":"invalid global counter shard detected; \\((.+?), (.+?), (.+?)\\) and \\((.+?), (.+?), (.+?)\\) differ only in count; will pick highest to self-heal on compaction","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/db/context/CounterContext.java","lineNumber":472,"sourceCode":"     * - DISJOINT for any two local shards\n     */\n    private Relationship compare(ContextState leftState, ContextState rightState)\n    {\n        long leftClock = leftState.getClock();\n        long leftCount = leftState.getCount();\n        long rightClock = rightState.getClock();\n        long rightCount = rightState.getCount();\n\n        if (leftState.isGlobal() || rightState.isGlobal())\n        {\n            if (leftState.isGlobal() && rightState.isGlobal())\n            {\n                if (leftClock == rightClock)\n                {\n                    // Can happen if an sstable gets lost and disk failure policy is set to 'best effort'\n                    if (leftCount != rightCount && CompactionManager.isCompactor(Thread.currentThread()))\n                    {\n                        logger.warn(\"invalid global counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in \"\n                                    + \"count; will pick highest to self-heal on compaction\",\n                                    leftState.getCounterId(), leftClock, leftCount,\n                                    rightState.getCounterId(), rightClock, rightCount);\n                    }\n\n                    if (leftCount > rightCount)\n                        return Relationship.GREATER_THAN;\n                    else if (leftCount == rightCount)\n                        return Relationship.EQUAL;\n                    else\n                        return Relationship.LESS_THAN;\n                }\n                else\n                {\n                    return leftClock > rightClock ? Relationship.GREATER_THAN : Relationship.LESS_THAN;\n                }\n            }\n            else // only one is global - keep that one","sourceCodeStart":454,"sourceCodeEnd":490,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/db/context/CounterContext.java#L454-L490","documentation":"CounterContext.compare logs this warning when it encounters two global counter shards with identical counterId and clock but different counts. This state is logically impossible under correct counter operation and usually indicates lost sstables or divergent writes; when detected on a compactor thread the comparison self-heals by deterministically picking the higher count, and the warning tells the operator data corruption was repaired in place.","triggerScenarios":"Reading/merging counter columns during compaction (isCompactor thread) where two shards match on id and clock but differ in count — e.g. an sstable was lost while disk_failure_policy is 'best_effort', allowing stale-differing counts to coexist; manual sstable deletion; disk corruption.","commonSituations":"Running with disk_failure_policy=best_effort after disk loss; restoring sstables from inconsistent backups; hardware issues corrupting individual sstables; counter tables on volumes with unreliable persistence.","solutions":["Investigate immediately: this indicates possible sstable loss or corruption. Check system logs around the time and verify sstable integrity (nodetool verify).","Avoid disk_failure_policy=best_effort; use stop/die so failures are surfaced rather than silently tolerated.","Run repairs on the affected counter table to converge replicas, and scrub/verify sstables.","Restore from backup if counts diverge materially; monitor for recurrence — repeated warnings imply ongoing storage problems."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// detect storage risk before it corrupts counters\nif (diskFailurePolicy.equals(\"best_effort\")) {\n    logger.warn(\"best_effort disk policy can produce invalid counter shards; consider 'stop' or 'die'\");\n}\nnodetoolVerifyScheduled(); // periodically run 'nodetool verify' on counter tables","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never use disk_failure_policy=best_effort for counter workloads.","Run regular nodetool verify/scrub on counter tables.","Only restore counter sstables from a single consistent backup snapshot.","Run repairs promptly after any storage incident.","Alert on this warning — it means corruption was silently healed."],"tags":["cassandra","counters","data-corruption","compaction","sstable"],"backgroundTag":"internal-invariant-violation","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}