What causes "Values for key {} differ: {} != {}" in apache/cassandra?

Read the logged key to identify the specific node whose entry diverged, then trace the transformation that changed it

Values for key differ: !=

Error message

Values for key {} differ: {} != {}

What it means

WARN log from the static Directory.dumpDiff(Logger, Map, Map) helper, emitted for each key present in both maps whose values are not Objects.equals. It provides per-key detail after a map-level divergence warning (peers, states, addresses, versions, etc.), showing exactly which entries differ. Keys present in only one map are logged by separate calls. Not an exception.

Solutions

Read the logged key to identify the specific node whose entry diverged, then trace the transformation that changed it
Replay or re-commit the corresponding TCM transformation on the stale view so the shared key's value converges
If the value should legitimately differ (different snapshot epochs), compare snapshots at the same epoch
Escalate to the Cassandra community if convergence fails without a partition, as it may indicate a metadata replication bug

Defensive patterns

Strategy: validation

Validate before calling

// Pre-compute divergent keys to act on them programmatically
Set<K> divergent = new HashSet<>(left.keySet());
divergent.retainAll(right.keySet());
divergent.removeIf(k -> java.util.Objects.equals(left.get(k), right.get(k)));

Prevention

Use the per-key diff output to drive targeted convergence rather than blanket restarts
Resolve the owning map-level divergence (peers/states/versions) first; per-key diffs are symptoms
Diff maps captured at the same epoch only
Automate alerting on divergent keys so they are fixed before they affect placement decisions

When it happens

Trigger: Any dumpDiff call on a diverging map where a shared key holds different values in left vs right map — e.g. a NodeId mapped to different NodeState, address, or version in the two snapshots being compared.

Common situations: Debugging exactly which nodes drive a Peers/States/Versions differ warning; post-mortem analysis of cluster metadata divergence; dtest failure output pinpointing the divergent node IDs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/7b2b2dc85bea0d31. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/tcm/membership/Directory.java:947

        {
            logger.warn("Versions differ: {} != {}", versions, other.versions);
            dumpDiff(logger, versions, other.versions);
        }
        if (!Objects.equals(addresses, other.addresses))
        {
            logger.warn("Addresses differ: {} != {}", addresses, other.addresses);
            dumpDiff(logger, addresses, other.addresses);
        }
    }

    public static <K, V> void dumpDiff(Logger logger, Map<K, V> l, Map<K, V> r)
    {
        for (K k : Sets.intersection(l.keySet(), r.keySet()))
        {
            V lv = l.get(k);
            V rv = r.get(k);
            if (!Objects.equals(lv, rv))
                logger.warn("Values for key {} differ: {} != {}", k, lv, rv);
        }
        for (K k : Sets.difference(l.keySet(), r.keySet()))
            logger.warn("Value for key {} is only present in the left set: {}", k, l.get(k));
        for (K k : Sets.difference(r.keySet(), l.keySet()))
            logger.warn("Value for key {} is only present in the right set: {}", k, r.get(k));

    }

    public static class RemovedNode implements Comparable<RemovedNode>
    {
        public final Epoch removedIn;
        public final NodeId id;
        public final InetAddressAndPort endpoint;

        public RemovedNode(Epoch removedIn, NodeId id, InetAddressAndPort endpoint)
        {
            this.removedIn = removedIn;
            this.id = id;

View on GitHub (pinned to 88fd0f6a0e)