invalid global counter shard detected; ({}, {}, {}) and ({},

Error message

invalid global counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in count; will pick highest to self-heal on compaction

What it means

CounterContext.compare logs this warning when it encounters two global counter shards with identical counterId and clock but different counts. This state is logically impossible under correct counter operation and usually indicates lost sstables or divergent writes; when detected on a compactor thread the comparison self-heals by deterministically picking the higher count, and the warning tells the operator data corruption was repaired in place.

Source

Thrown at src/java/org/apache/cassandra/db/context/CounterContext.java:472

     * - DISJOINT for any two local shards
     */
    private Relationship compare(ContextState leftState, ContextState rightState)
    {
        long leftClock = leftState.getClock();
        long leftCount = leftState.getCount();
        long rightClock = rightState.getClock();
        long rightCount = rightState.getCount();

        if (leftState.isGlobal() || rightState.isGlobal())
        {
            if (leftState.isGlobal() && rightState.isGlobal())
            {
                if (leftClock == rightClock)
                {
                    // Can happen if an sstable gets lost and disk failure policy is set to 'best effort'
                    if (leftCount != rightCount && CompactionManager.isCompactor(Thread.currentThread()))
                    {
                        logger.warn("invalid global counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in "
                                    + "count; will pick highest to self-heal on compaction",
                                    leftState.getCounterId(), leftClock, leftCount,
                                    rightState.getCounterId(), rightClock, rightCount);
                    }

                    if (leftCount > rightCount)
                        return Relationship.GREATER_THAN;
                    else if (leftCount == rightCount)
                        return Relationship.EQUAL;
                    else
                        return Relationship.LESS_THAN;
                }
                else
                {
                    return leftClock > rightClock ? Relationship.GREATER_THAN : Relationship.LESS_THAN;
                }
            }
            else // only one is global - keep that one

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Investigate immediately: this indicates possible sstable loss or corruption. Check system logs around the time and verify sstable integrity (nodetool verify).
Avoid disk_failure_policy=best_effort; use stop/die so failures are surfaced rather than silently tolerated.
Run repairs on the affected counter table to converge replicas, and scrub/verify sstables.
Restore from backup if counts diverge materially; monitor for recurrence — repeated warnings imply ongoing storage problems.

Defensive patterns

Strategy: validation

Validate before calling

// detect storage risk before it corrupts counters
if (diskFailurePolicy.equals("best_effort")) {
    logger.warn("best_effort disk policy can produce invalid counter shards; consider 'stop' or 'die'");
}
nodetoolVerifyScheduled(); // periodically run 'nodetool verify' on counter tables

Prevention

Never use disk_failure_policy=best_effort for counter workloads.
Run regular nodetool verify/scrub on counter tables.
Only restore counter sstables from a single consistent backup snapshot.
Run repairs promptly after any storage incident.
Alert on this warning — it means corruption was silently healed.

When it happens

Trigger: Reading/merging counter columns during compaction (isCompactor thread) where two shards match on id and clock but differ in count — e.g. an sstable was lost while disk_failure_policy is 'best_effort', allowing stale-differing counts to coexist; manual sstable deletion; disk corruption.

Common situations: Running with disk_failure_policy=best_effort after disk loss; restoring sstables from inconsistent backups; hardware issues corrupting individual sstables; counter tables on volumes with unreliable persistence.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/137c705b35ae8d80. Report an issue: GitHub.