invalid remote counter shard detected; ({}, {}, {}) and ({},

Error message

invalid remote counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in count; will pick highest to self-heal on compaction

What it means

Same self-healing logic as the global shard case, but for remote counter shards: two shards with the same counterId and clock but different counts. Per the code comment this should never happen; when it does on a compactor thread, the highest count is deterministically chosen and the occurrence is logged so operators know something went wrong.

Source

Thrown at src/java/org/apache/cassandra/db/context/CounterContext.java:513

        if (leftState.isLocal() || rightState.isLocal())
        {
            // Local id and at least one is a local shard.
            if (leftState.isLocal() && rightState.isLocal())
                return Relationship.DISJOINT;
            else // only one is local - keep that one
                return leftState.isLocal() ? Relationship.GREATER_THAN : Relationship.LESS_THAN;
        }

        // both are remote shards
        if (leftClock == rightClock)
        {
            // We should never see non-local shards w/ same id+clock but different counts. However, if we do
            // we should "heal" the problem by being deterministic in our selection of shard - and
            // log the occurrence so that the operator will know something is wrong.
            if (leftCount != rightCount && CompactionManager.isCompactor(Thread.currentThread()))
            {
                logger.warn("invalid remote counter shard detected; ({}, {}, {}) and ({}, {}, {}) differ only in "
                            + "count; will pick highest to self-heal on compaction",
                            leftState.getCounterId(), leftClock, leftCount,
                            rightState.getCounterId(), rightClock, rightCount);
            }

            if (leftCount > rightCount)
                return Relationship.GREATER_THAN;
            else if (leftCount == rightCount)
                return Relationship.EQUAL;
            else
                return Relationship.LESS_THAN;
        }
        else
        {
            if ((leftClock >= 0 && rightClock > 0 && leftClock >= rightClock)
                    || (leftClock < 0 && (rightClock > 0 || leftClock < rightClock)))
                return Relationship.GREATER_THAN;
            else

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Run nodetool verify/scrub on the node to detect corrupt sstables and run repair on the counter table.
Investigate backup/restore procedures — do not mix sstables from different restore points.
Replace best_effort disk policy with stop or die to catch storage failures deterministically.
If counts are business-critical, compare against replicas and consider re-writing the counter from application-level data.

Defensive patterns

Strategy: validation

Validate before calling

// audit restore hygiene for counter tables
if (restoredSstablesMixedSnapshots) {
    throw new IllegalStateException("Refusing mixed-snapshot sstable restore for counter table");
}

Prevention

Do not copy sstables manually between nodes or DCs.
Restore counter tables only from single, complete snapshots.
Use stop/die disk failure policy to surface storage failures.
Run nodetool verify after any suspicious storage event.
Alert on this warning — it signals impossible counter state was healed.

When it happens

Trigger: Compaction (isCompactor thread) comparing remote (non-local) counter shards with equal id+clock but unequal counts — caused by lost/corrupt sstables, improper manual sstable manipulation, or restored data that is inconsistent within a single sstable set.

Common situations: Restoring partial backups of counter tables; sstable corruption; copying sstables between nodes/datacenters manually; best_effort disk failure policy masking data loss.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/e3136bf165174bfc. Report an issue: GitHub.